[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:29 dist-test-slave-qfmw krb5kdc[13274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:29 dist-test-slave-qfmw krb5kdc[13274](info): set up 2 sockets
Sep 26 02:19:29 dist-test-slave-qfmw krb5kdc[13274](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 02:19:30.321991 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:19:30 dist-test-slave-qfmw krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:19:30.507016 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:43247
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:30.865605 13287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:30.865926 13287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:30.866111 13287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:30.873669 13287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:30.873793 13287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:30.873906 13287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:30.873991 13287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:30.874063 13287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:30.884235 13287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:30.885645 13287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:30.887018 13287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:30.900334 13295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:30.900425 13292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:30.900466 13293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:30.901189 13287 server_base.cc:1034] running on GCE node
I20260926 02:19:30.902020 13287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:30.903631 13287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:30.904863 13287 hybrid_clock.cc:648] HybridClock initialized: now 1790389170904802 us; error 69 us; skew 500 ppm
I20260926 02:19:30.905222 13287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:30 dist-test-slave-qfmw krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389170, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:30.908361 13287 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:19:30.909842 13287 webserver.cc:460] Webserver started at http://127.12.244.254:43451/ using document root <none> and password file <none>
I20260926 02:19:30.910552 13287 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:30.910702 13287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:30.911054 13287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:30.914302 13287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "27530d7ae26a4aa28b385fe8b4aca7ea"
format_stamp: "Formatted at 2026-09-26 02:19:30 on dist-test-slave-qfmw"
server_key: "1c7c181db6c7e09872d90365f7cde866"
server_key_iv: "40184539995d939d63bd475da6a2e12e"
server_key_version: "encryptionkey@0"
I20260926 02:19:30.915300 13287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "27530d7ae26a4aa28b385fe8b4aca7ea"
format_stamp: "Formatted at 2026-09-26 02:19:30 on dist-test-slave-qfmw"
server_key: "1c7c181db6c7e09872d90365f7cde866"
server_key_iv: "40184539995d939d63bd475da6a2e12e"
server_key_version: "encryptionkey@0"
I20260926 02:19:30.920992 13287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:19:30.925524 13302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:30.927217 13287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:19:30.927440 13287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "27530d7ae26a4aa28b385fe8b4aca7ea"
format_stamp: "Formatted at 2026-09-26 02:19:30 on dist-test-slave-qfmw"
server_key: "1c7c181db6c7e09872d90365f7cde866"
server_key_iv: "40184539995d939d63bd475da6a2e12e"
server_key_version: "encryptionkey@0"
I20260926 02:19:30.927675 13287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:30.952803 13287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:30.958026 13287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:30.958382 13287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:30.978111 13287 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:43247
I20260926 02:19:30.978143 13353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:43247 every 8 connection(s)
I20260926 02:19:30.979650 13287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:30.983660 13354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:30.985571 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 13287
I20260926 02:19:30.985924 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:19:30.988514 13267 external_mini_cluster.cc:1428] Setting key 365632379cedcab258f3294fdde7c24c
I20260926 02:19:30.997172 13354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea: Bootstrap starting.
I20260926 02:19:31.000312 13354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:31.001376 13354 log.cc:824] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:31.004833 13354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea: No bootstrap required, opened a new log
I20260926 02:19:31.014313 13354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43247 } }
I20260926 02:19:31.014768 13354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:31.014918 13354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27530d7ae26a4aa28b385fe8b4aca7ea, State: Initialized, Role: FOLLOWER
I20260926 02:19:31.015642 13354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43247 } }
I20260926 02:19:31.015906 13354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:31.016049 13354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:31.016250 13354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:31.018812 13354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43247 } }
I20260926 02:19:31.019397 13354 leader_election.cc:302] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27530d7ae26a4aa28b385fe8b4aca7ea; no voters:
I20260926 02:19:31.020040 13354 leader_election.cc:288] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:31.020159 13359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:31.021423 13359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [term 1 LEADER]: Becoming Leader. State: Replica: 27530d7ae26a4aa28b385fe8b4aca7ea, State: Running, Role: LEADER
I20260926 02:19:31.022046 13359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43247 } }
Sep 26 02:19:31 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:31.022635 13354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:31.024842 13361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27530d7ae26a4aa28b385fe8b4aca7ea. Latest consensus state: current_term: 1 leader_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43247 } } }
I20260926 02:19:31.025200 13361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:31.025851 13360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27530d7ae26a4aa28b385fe8b4aca7ea" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43247 } } }
I20260926 02:19:31.026134 13360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:31.026677 13364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:31.039151 13364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:31.041673 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:30.993396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.12.244.254:43247)
0926 02:19:30.994420 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:30.994441 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:30.995895 (+ 1454us) server_negotiation.cc:407] Connection header received
0926 02:19:31.006259 (+ 10364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:31.006296 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:31.006873 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:31.007419 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:31.010707 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:31.013035 (+ 2328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:31.015408 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:31.016378 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:31.016617 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:31.024338 (+ 7721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:31.024354 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:31.024385 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:31.024463 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:31.034075 (+ 9612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:31.035634 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:31.035644 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:31.035654 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:31.035926 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:31.036628 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:31.036633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:31.036636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:31.037490 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:31.037829 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:31.039446 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0926 02:19:31.040010 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":427,"threads_started":1}
I20260926 02:19:31.057325 13364 catalog_manager.cc:1347] Generated new cluster ID: 737fd4f794ce47beb585965fd2473274
I20260926 02:19:31.057471 13364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:31.078291 13364 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:31.079792 13364 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:31.099016 13364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27530d7ae26a4aa28b385fe8b4aca7ea: Generated new TSK 0
I20260926 02:19:31.100237 13364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:19:31.158631 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43247
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:31.531507 13380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:31.531896 13380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:31.532081 13380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:31.539788 13380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:31.539911 13380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:31.540091 13380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:19:31.551532 13380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43247
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:31.553124 13380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:31.554754 13380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:31.571197 13388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:31.571169 13386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:31.571255 13385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:31.571950 13380 server_base.cc:1034] running on GCE node
I20260926 02:19:31.572610 13380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:31.573599 13380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:31.574846 13380 hybrid_clock.cc:648] HybridClock initialized: now 1790389171574771 us; error 60 us; skew 500 ppm
I20260926 02:19:31.575279 13380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:31 dist-test-slave-qfmw krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389171, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:31.578482 13380 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:19:31.580168 13380 webserver.cc:460] Webserver started at http://127.12.244.193:36631/ using document root <none> and password file <none>
I20260926 02:19:31.580878 13380 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:31.581034 13380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:31.581347 13380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:31.584517 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "a12e24d3b5f544cebc32c707ff53af5e"
format_stamp: "Formatted at 2026-09-26 02:19:31 on dist-test-slave-qfmw"
server_key: "51679532f51568335b05ea27b7711455"
server_key_iv: "4e81e7ceaab9830251f084a32091bd73"
server_key_version: "encryptionkey@0"
I20260926 02:19:31.585340 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "a12e24d3b5f544cebc32c707ff53af5e"
format_stamp: "Formatted at 2026-09-26 02:19:31 on dist-test-slave-qfmw"
server_key: "51679532f51568335b05ea27b7711455"
server_key_iv: "4e81e7ceaab9830251f084a32091bd73"
server_key_version: "encryptionkey@0"
I20260926 02:19:31.591157 13380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:19:31.595223 13395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:31.596777 13380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:31.596980 13380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "a12e24d3b5f544cebc32c707ff53af5e"
format_stamp: "Formatted at 2026-09-26 02:19:31 on dist-test-slave-qfmw"
server_key: "51679532f51568335b05ea27b7711455"
server_key_iv: "4e81e7ceaab9830251f084a32091bd73"
server_key_version: "encryptionkey@0"
I20260926 02:19:31.597213 13380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:31.619266 13380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:31.624147 13380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:31.624461 13380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:31.625563 13380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:31.626950 13380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:31.627063 13380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:31.627223 13380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:31.627290 13380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:31.661098 13380 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:35777
I20260926 02:19:31.661144 13507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:35777 every 8 connection(s)
I20260926 02:19:31.662433 13380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:19:31.669680 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 13380
I20260926 02:19:31.669965 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:19:31.670675 13267 external_mini_cluster.cc:1428] Setting key 7b4dbf18df3f4219712fc00d9d5b3e7f
Sep 26 02:19:31 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389171, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:31.685176 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:31.665099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:56963 (local address 127.12.244.254:43247)
0926 02:19:31.665530 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:31.665535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:31.666649 (+ 1114us) server_negotiation.cc:407] Connection header received
0926 02:19:31.668565 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:31.668570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:31.668639 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:31.668743 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:31.670471 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:31.671542 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:31.673858 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:31.674261 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:31.674380 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:31.678769 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:31.678775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:31.678778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:31.678813 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:31.681438 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:31.682501 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:31.682505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:31.682508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:31.682624 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:31.683180 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:31.683182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:31.683185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:31.683448 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:31.683561 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:31.684656 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 02:19:31.684908 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":221,"threads_started":1}
I20260926 02:19:31.686849 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:31.665761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43247 (local address 127.12.244.193:56963)
0926 02:19:31.666454 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:31.666492 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:31.668252 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:31.668961 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:31.668969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:31.669526 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:31.670272 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:31.670293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:31.671763 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:31.671767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:31.673697 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:31.673714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:31.674515 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:31.674525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:31.674905 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:31.675758 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:31.675795 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:31.678500 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:31.681606 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:31.681620 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:31.681642 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:31.682350 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:31.682737 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:31.682745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:31.682752 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:31.683018 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:31.683582 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:31.683593 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:31.684164 (+ 571us) client_negotiation.cc:769] Sending connection context
0926 02:19:31.684531 (+ 367us) client_negotiation.cc:240] Negotiation successful
0926 02:19:31.685223 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":263,"threads_started":1}
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:19:31.688603 13508 heartbeater.cc:347] Connected to a master server at 127.12.244.254:43247
I20260926 02:19:31.688942 13508 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:31.689659 13508 heartbeater.cc:511] Master 127.12.244.254:43247 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:19:31.691887 13319 ts_manager.cc:194] Registered new tserver with Master: a12e24d3b5f544cebc32c707ff53af5e (127.12.244.193:35777)
I20260926 02:19:31.693377 13319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:56963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:19:31.712395 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43247
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:32.081333 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:32.081646 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:32.081743 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:32.089264 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:32.089407 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:32.089541 13514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:19:32.100427 13514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43247
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:32.101800 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:32.103626 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:32.117254 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:32.117282 13519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:32.117287 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:32.118091 13514 server_base.cc:1034] running on GCE node
I20260926 02:19:32.118762 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:32.119905 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:32.121117 13514 hybrid_clock.cc:648] HybridClock initialized: now 1790389172121065 us; error 49 us; skew 500 ppm
I20260926 02:19:32.121456 13514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:32.125003 13514 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:19:32.126732 13514 webserver.cc:460] Webserver started at http://127.12.244.194:44755/ using document root <none> and password file <none>
I20260926 02:19:32.127553 13514 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:32.127707 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:32.128043 13514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:32.131275 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "6db1498ea33744dc947d04befa99abfa"
format_stamp: "Formatted at 2026-09-26 02:19:32 on dist-test-slave-qfmw"
server_key: "d6baead7c03599e9455a78608a18f118"
server_key_iv: "ea08b14dee7b1aee492dd63189b58694"
server_key_version: "encryptionkey@0"
I20260926 02:19:32.132185 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "6db1498ea33744dc947d04befa99abfa"
format_stamp: "Formatted at 2026-09-26 02:19:32 on dist-test-slave-qfmw"
server_key: "d6baead7c03599e9455a78608a18f118"
server_key_iv: "ea08b14dee7b1aee492dd63189b58694"
server_key_version: "encryptionkey@0"
I20260926 02:19:32.138192 13514 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 02:19:32.142298 13529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:32.144073 13514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:32.144279 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "6db1498ea33744dc947d04befa99abfa"
format_stamp: "Formatted at 2026-09-26 02:19:32 on dist-test-slave-qfmw"
server_key: "d6baead7c03599e9455a78608a18f118"
server_key_iv: "ea08b14dee7b1aee492dd63189b58694"
server_key_version: "encryptionkey@0"
I20260926 02:19:32.144569 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:32.166337 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:32.171474 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:32.171800 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:32.173326 13514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:32.174907 13514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:32.175040 13514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:32.175209 13514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:32.175273 13514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:32.209362 13514 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:38793
I20260926 02:19:32.209412 13641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:38793 every 8 connection(s)
I20260926 02:19:32.210721 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:19:32.212661 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 13514
I20260926 02:19:32.212878 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:19:32.213474 13267 external_mini_cluster.cc:1428] Setting key fc90c0fdea1fb3c36f70524aa032db32
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
I20260926 02:19:32.237325 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:32.213880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:49085 (local address 127.12.244.254:43247)
0926 02:19:32.214342 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:32.214347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:32.216088 (+ 1741us) server_negotiation.cc:407] Connection header received
0926 02:19:32.219227 (+ 3139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:32.219232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:32.219305 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:32.219448 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:32.221675 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.222720 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.224642 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.225435 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.225588 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:32.229569 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:32.229578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:32.229585 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:32.229637 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:32.233471 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.234304 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.234309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.234311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.234462 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.235043 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.235048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.235050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.235538 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:32.235696 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:32.236556 (+ 860us) server_negotiation.cc:299] Negotiation successful
0926 02:19:32.236842 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":268,"threads_started":1}
I20260926 02:19:32.238026 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:32.214807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43247 (local address 127.12.244.194:49085)
0926 02:19:32.215903 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:32.215962 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:32.218808 (+ 2846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:32.219768 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:32.219784 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:32.220715 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:32.221448 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:32.221468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.222896 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.222900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:32.224484 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:32.224498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.225670 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.225675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:32.225885 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:32.226653 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:32.226686 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:32.229262 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:32.233651 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:32.233660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:32.233676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:32.234150 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:32.234606 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:32.234610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:32.234612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:32.234834 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:32.235699 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:32.235710 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:32.236136 (+ 426us) client_negotiation.cc:769] Sending connection context
0926 02:19:32.236397 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 02:19:32.236835 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":500,"threads_started":1}
I20260926 02:19:32.239681 13642 heartbeater.cc:347] Connected to a master server at 127.12.244.254:43247
I20260926 02:19:32.240025 13642 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:32.240806 13642 heartbeater.cc:511] Master 127.12.244.254:43247 requested a full tablet report, sending...
I20260926 02:19:32.242643 13319 ts_manager.cc:194] Registered new tserver with Master: 6db1498ea33744dc947d04befa99abfa (127.12.244.194:38793)
I20260926 02:19:32.243593 13319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:49085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:19:32.259744 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43247
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:32.651407 13648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:32.651834 13648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:32.652067 13648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:32.662395 13648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:32.662640 13648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:32.662885 13648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:19:32.676252 13648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43247
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:32.677948 13648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:32.679723 13648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 02:19:32.697368 13508 heartbeater.cc:503] Master 127.12.244.254:43247 was elected leader, sending a full tablet report...
W20260926 02:19:32.701853 13654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:32.703926 13653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:32.704028 13656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:32.704974 13648 server_base.cc:1034] running on GCE node
I20260926 02:19:32.705968 13648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:32.707468 13648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:32.708761 13648 hybrid_clock.cc:648] HybridClock initialized: now 1790389172708658 us; error 59 us; skew 500 ppm
I20260926 02:19:32.709200 13648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:32.713943 13648 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:19:32.716151 13648 webserver.cc:460] Webserver started at http://127.12.244.195:37755/ using document root <none> and password file <none>
I20260926 02:19:32.717253 13648 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:32.717530 13648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:32.717957 13648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:32.721894 13648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "8c0b329168c04c86b9a1f5d4e0b369af"
format_stamp: "Formatted at 2026-09-26 02:19:32 on dist-test-slave-qfmw"
server_key: "69a8a7c8c7e522438f8f67a476f1a30a"
server_key_iv: "dc228c48c253b1e518b18aea52742ad4"
server_key_version: "encryptionkey@0"
I20260926 02:19:32.723135 13648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "8c0b329168c04c86b9a1f5d4e0b369af"
format_stamp: "Formatted at 2026-09-26 02:19:32 on dist-test-slave-qfmw"
server_key: "69a8a7c8c7e522438f8f67a476f1a30a"
server_key_iv: "dc228c48c253b1e518b18aea52742ad4"
server_key_version: "encryptionkey@0"
I20260926 02:19:32.730618 13648 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 02:19:32.736627 13663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:32.738652 13648 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:19:32.738919 13648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "8c0b329168c04c86b9a1f5d4e0b369af"
format_stamp: "Formatted at 2026-09-26 02:19:32 on dist-test-slave-qfmw"
server_key: "69a8a7c8c7e522438f8f67a476f1a30a"
server_key_iv: "dc228c48c253b1e518b18aea52742ad4"
server_key_version: "encryptionkey@0"
I20260926 02:19:32.739379 13648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:32.758880 13648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:32.765414 13648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:32.765820 13648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:32.767458 13648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:32.769598 13648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:32.769714 13648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:32.769994 13648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:32.770102 13648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:32.814004 13648 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:41877
I20260926 02:19:32.814057 13775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:41877 every 8 connection(s)
I20260926 02:19:32.815450 13648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:19:32.820659 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 13648
I20260926 02:19:32.820888 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:19:32.821494 13267 external_mini_cluster.cc:1428] Setting key 43828de2edcf0869a5a54d8e5cdb8920
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:32.840747 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:32.818433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:43179 (local address 127.12.244.254:43247)
0926 02:19:32.818905 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:32.818910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:32.820272 (+ 1362us) server_negotiation.cc:407] Connection header received
0926 02:19:32.823284 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:32.823288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:32.823433 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:32.823581 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:32.825941 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.826835 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.828759 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.829497 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.829670 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:32.833754 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:32.833760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:32.833764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:32.833800 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:32.836957 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.838107 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.838111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.838114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.838270 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.838849 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.838854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.838857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.839318 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:32.839431 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:32.840244 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 02:19:32.840460 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260926 02:19:32.841749 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:32.819216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43247 (local address 127.12.244.195:43179)
0926 02:19:32.820023 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:32.820086 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:32.822873 (+ 2787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:32.823972 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:32.823987 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:32.824859 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:32.825641 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:32.825676 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.827021 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.827025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:32.828560 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:32.828575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.829814 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.829820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:32.830115 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:32.830837 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:32.830924 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:32.833505 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:32.837245 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:32.837256 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:32.837272 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:32.837975 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:32.838436 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:32.838442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:32.838446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:32.838717 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:32.839443 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:32.839450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:32.839821 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 02:19:32.840119 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 02:19:32.840539 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":308,"threads_started":1}
I20260926 02:19:32.843559 13776 heartbeater.cc:347] Connected to a master server at 127.12.244.254:43247
I20260926 02:19:32.843904 13776 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:32.844925 13776 heartbeater.cc:511] Master 127.12.244.254:43247 requested a full tablet report, sending...
I20260926 02:19:32.847303 13319 ts_manager.cc:194] Registered new tserver with Master: 8c0b329168c04c86b9a1f5d4e0b369af (127.12.244.195:41877)
I20260926 02:19:32.848258 13319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:43179
I20260926 02:19:32.852505 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:19:32.900900 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:32.883629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.12.244.193:35777)
0926 02:19:32.884374 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:32.884382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:32.884412 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:19:32.884607 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:32.884615 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:32.884844 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:32.885042 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:32.886471 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.887634 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.888917 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.889496 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.889597 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:32.894400 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:32.894416 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:32.894432 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:32.894479 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:32.897550 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.898490 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.898499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.898501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.898641 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.899229 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.899233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.899235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.899626 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:32.899894 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:32.900279 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 02:19:32.900495 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":465,"threads_started":1}
W20260926 02:19:32.902184 13423 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45064
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:19:32 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:19:32.949955 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:32.934144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.12.244.193:35777)
0926 02:19:32.934369 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:32.934375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:32.934535 (+ 160us) server_negotiation.cc:407] Connection header received
0926 02:19:32.934683 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:32.934687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:32.934787 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:32.934884 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:32.936398 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.937098 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.938521 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.938978 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.939162 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:32.943829 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:32.943835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:32.943839 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:32.943880 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:32.946706 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.947548 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.947553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.947556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.947704 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.948325 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.948329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.948331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.948634 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:32.948837 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:32.949305 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 02:19:32.949511 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 02:19:32.951777 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 13380
I20260926 02:19:32.966933 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:32.952890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.12.244.193:35777)
0926 02:19:32.953122 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:32.953130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:32.953391 (+ 261us) server_negotiation.cc:407] Connection header received
0926 02:19:32.953602 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:32.953606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:32.953711 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:32.953827 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:32.955160 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.955863 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.957179 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:32.957608 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:32.957747 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:32.960374 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:32.960380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:32.960383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:32.960421 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:32.964097 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.964896 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.964902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.964906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.965075 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:32.965481 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:32.965488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:32.965492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:32.965776 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:32.966031 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:32.966338 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 02:19:32.966586 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:19:32.967756 13503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:33.053434 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 13380
I20260926 02:19:33.070860 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 13514
Sep 26 02:19:33 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:19:33.088474 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:33.071894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.12.244.194:38793)
0926 02:19:33.072435 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:33.072442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:33.072466 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:19:33.072525 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:33.072531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:33.072762 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:33.072951 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:33.074202 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:33.075479 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:33.076779 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:33.077369 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:33.077517 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:33.082008 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:33.082023 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:33.082042 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:33.082102 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:33.085207 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:33.086025 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:33.086033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:33.086035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:33.086172 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:33.086683 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:33.086688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:33.086690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:33.087087 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:33.087394 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:33.087768 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 02:19:33.088098 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":291,"threads_started":1}
I20260926 02:19:33.089982 13637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:33.174803 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 13514
I20260926 02:19:33.197821 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 13648
Sep 26 02:19:33 dist-test-slave-qfmw krb5kdc[13274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:19:33.216205 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:33.199024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.12.244.195:41877)
0926 02:19:33.199706 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:33.199713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:33.199743 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:19:33.199808 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:33.199815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:33.200053 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:33.200249 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:33.201668 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:33.202824 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:33.204305 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:33.204945 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:33.205113 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:33.209875 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:33.209891 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:33.209908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:33.209956 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:33.213089 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:33.213909 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:33.213917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:33.213919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:33.214111 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:33.214564 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:33.214568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:33.214570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:33.214943 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:33.215410 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:33.215681 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 02:19:33.215897 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":391,"threads_started":1}
I20260926 02:19:33.217579 13771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:33.302376 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 13648
I20260926 02:19:33.319056 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 13287
I20260926 02:19:33.320356 13349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:33.427949 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 13287
2026-09-26T02:19:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4233 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:33 dist-test-slave-qfmw krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:33 dist-test-slave-qfmw krb5kdc[13798](info): set up 2 sockets
Sep 26 02:19:33 dist-test-slave-qfmw krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260926 02:19:34.521593 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:19:34 dist-test-slave-qfmw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:19:34.710356 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:38115
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:38115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:35.040809 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:35.041196 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:35.041369 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:35.048908 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:35.049077 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:35.049192 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:35.049284 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:35.049363 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:35.059964 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:38115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:35.061391 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:35.062778 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:35.075304 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:35.075351 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:35.075387 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:35.076530 13811 server_base.cc:1034] running on GCE node
I20260926 02:19:35.077539 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:35.079303 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:35.080571 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790389175080531 us; error 49 us; skew 500 ppm
I20260926 02:19:35.080982 13811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:35 dist-test-slave-qfmw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389175, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:35.094535 13811 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:19:35.096481 13811 webserver.cc:460] Webserver started at http://127.12.244.254:39115/ using document root <none> and password file <none>
I20260926 02:19:35.097309 13811 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:35.097476 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:35.097782 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:35.101099 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "357349bfd9374886ac3d2a850507cf7f"
format_stamp: "Formatted at 2026-09-26 02:19:35 on dist-test-slave-qfmw"
server_key: "42478d43b1b7fb6a0f7ccd9d3d07646b"
server_key_iv: "a30d1fdc973dd6a69aeb3b7fc803591c"
server_key_version: "encryptionkey@0"
I20260926 02:19:35.101899 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "357349bfd9374886ac3d2a850507cf7f"
format_stamp: "Formatted at 2026-09-26 02:19:35 on dist-test-slave-qfmw"
server_key: "42478d43b1b7fb6a0f7ccd9d3d07646b"
server_key_iv: "a30d1fdc973dd6a69aeb3b7fc803591c"
server_key_version: "encryptionkey@0"
I20260926 02:19:35.107864 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 02:19:35.111699 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:35.113204 13811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:19:35.113396 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "357349bfd9374886ac3d2a850507cf7f"
format_stamp: "Formatted at 2026-09-26 02:19:35 on dist-test-slave-qfmw"
server_key: "42478d43b1b7fb6a0f7ccd9d3d07646b"
server_key_iv: "a30d1fdc973dd6a69aeb3b7fc803591c"
server_key_version: "encryptionkey@0"
I20260926 02:19:35.113626 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:35.135993 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:35.140664 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:35.140980 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:35.160476 13811 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:38115
I20260926 02:19:35.160506 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:38115 every 8 connection(s)
I20260926 02:19:35.161970 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:35.165964 13878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:35.169193 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 13811
I20260926 02:19:35.169454 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:19:35.170079 13267 external_mini_cluster.cc:1428] Setting key 686da7699b9dd1402556e7b7172d4e41
I20260926 02:19:35.180177 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f: Bootstrap starting.
I20260926 02:19:35.184171 13878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:35.185240 13878 log.cc:824] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:19:35 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:35.188616 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f: No bootstrap required, opened a new log
I20260926 02:19:35.195847 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:35.172631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.12.244.254:38115)
0926 02:19:35.173469 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:35.173486 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:35.173534 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:19:35.175601 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:35.175645 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:35.176291 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:35.176797 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:35.178130 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:35.180174 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:35.181779 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:35.182882 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:35.183134 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:35.187594 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:35.187603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:35.187620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:35.187670 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:35.190961 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:35.191986 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:35.191994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:35.192001 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:35.192188 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:35.192670 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:35.192674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:35.192676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:35.193316 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:35.193558 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:35.194158 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 02:19:35.194635 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":357,"threads_started":1}
I20260926 02:19:35.197577 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357349bfd9374886ac3d2a850507cf7f" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 38115 } }
I20260926 02:19:35.197983 13878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:35.198169 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357349bfd9374886ac3d2a850507cf7f, State: Initialized, Role: FOLLOWER
I20260926 02:19:35.198770 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357349bfd9374886ac3d2a850507cf7f" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 38115 } }
I20260926 02:19:35.199059 13878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:35.199239 13878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:35.199405 13878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:35.201581 13878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357349bfd9374886ac3d2a850507cf7f" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 38115 } }
I20260926 02:19:35.202073 13878 leader_election.cc:302] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 357349bfd9374886ac3d2a850507cf7f; no voters:
I20260926 02:19:35.202643 13878 leader_election.cc:288] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:35.202807 13883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:35.203596 13883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [term 1 LEADER]: Becoming Leader. State: Replica: 357349bfd9374886ac3d2a850507cf7f, State: Running, Role: LEADER
I20260926 02:19:35.204171 13883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357349bfd9374886ac3d2a850507cf7f" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 38115 } }
I20260926 02:19:35.204991 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:35.207006 13885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 357349bfd9374886ac3d2a850507cf7f. Latest consensus state: current_term: 1 leader_uuid: "357349bfd9374886ac3d2a850507cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357349bfd9374886ac3d2a850507cf7f" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 38115 } } }
I20260926 02:19:35.207281 13884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "357349bfd9374886ac3d2a850507cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357349bfd9374886ac3d2a850507cf7f" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 38115 } } }
I20260926 02:19:35.207491 13885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:35.207587 13884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:35.210515 13889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:35.216046 13889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:35.226104 13889 catalog_manager.cc:1347] Generated new cluster ID: 839843a5280341ec95b5e83e6fabef91
I20260926 02:19:35.226262 13889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:35.237866 13889 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:35.239632 13889 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:35.259228 13889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 357349bfd9374886ac3d2a850507cf7f: Generated new TSK 0
I20260926 02:19:35.260171 13889 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:19:35.312290 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:38115
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:35.677018 13904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:35.677320 13904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:35.677479 13904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:35.685252 13904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:35.685385 13904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:35.685593 13904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:19:35.696099 13904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:38115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:35.697583 13904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:35.698971 13904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:35.713384 13909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:35.713428 13912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:35.713312 13910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:35.714583 13904 server_base.cc:1034] running on GCE node
I20260926 02:19:35.715373 13904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:35.716506 13904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:35.717737 13904 hybrid_clock.cc:648] HybridClock initialized: now 1790389175717690 us; error 50 us; skew 500 ppm
I20260926 02:19:35.718129 13904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:35 dist-test-slave-qfmw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389175, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:35.721817 13904 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:19:35.723634 13904 webserver.cc:460] Webserver started at http://127.12.244.193:35283/ using document root <none> and password file <none>
I20260926 02:19:35.724390 13904 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:35.724552 13904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:35.724892 13904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:35.728078 13904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "87574fecf2624703b13e3585ac8378c4"
format_stamp: "Formatted at 2026-09-26 02:19:35 on dist-test-slave-qfmw"
server_key: "ad797b0ff89d2f1d9d9ee58149e68d8c"
server_key_iv: "a758607a708064e93f76ff0955058cd6"
server_key_version: "encryptionkey@0"
I20260926 02:19:35.728940 13904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "87574fecf2624703b13e3585ac8378c4"
format_stamp: "Formatted at 2026-09-26 02:19:35 on dist-test-slave-qfmw"
server_key: "ad797b0ff89d2f1d9d9ee58149e68d8c"
server_key_iv: "a758607a708064e93f76ff0955058cd6"
server_key_version: "encryptionkey@0"
I20260926 02:19:35.734754 13904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:19:35.738685 13919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:35.740352 13904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:35.740558 13904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "87574fecf2624703b13e3585ac8378c4"
format_stamp: "Formatted at 2026-09-26 02:19:35 on dist-test-slave-qfmw"
server_key: "ad797b0ff89d2f1d9d9ee58149e68d8c"
server_key_iv: "a758607a708064e93f76ff0955058cd6"
server_key_version: "encryptionkey@0"
I20260926 02:19:35.740775 13904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:35.764938 13904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:35.769667 13904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:35.770049 13904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:35.771255 13904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:35.772722 13904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:35.772830 13904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:35.772982 13904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:35.773056 13904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:35.807533 13904 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:42063
I20260926 02:19:35.807621 14031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:42063 every 8 connection(s)
I20260926 02:19:35.808835 13904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:19:35.815378 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 13904
I20260926 02:19:35.815624 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:19:35.816499 13267 external_mini_cluster.cc:1428] Setting key 87535125d2b70537b7b4cfab63cca7a6
Sep 26 02:19:35 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389175, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:35.834118 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:35.811627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:37023 (local address 127.12.244.254:38115)
0926 02:19:35.812091 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:35.812096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:35.813262 (+ 1166us) server_negotiation.cc:407] Connection header received
0926 02:19:35.815721 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:35.815727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:35.815833 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:35.815932 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:35.818211 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:35.819240 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:35.822134 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:35.822578 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:35.822694 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:35.826653 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:35.826663 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:35.826671 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:35.826722 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:35.830369 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:35.831409 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:35.831418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:35.831425 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:35.831605 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:35.832051 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:35.832060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:35.832066 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:35.832495 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:35.832710 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:35.833449 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 02:19:35.833715 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":258,"threads_started":1}
I20260926 02:19:35.834959 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:35.812273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:38115 (local address 127.12.244.193:37023)
0926 02:19:35.813065 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:35.813110 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:35.815271 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:35.816291 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:35.816307 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:35.817223 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:35.818021 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:35.818047 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:35.819434 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:35.819440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:35.821917 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:35.821939 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:35.822848 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:35.822857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:35.823230 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:35.824056 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:35.824096 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:35.826366 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:35.830599 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:35.830610 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:35.830632 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:35.831237 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:35.831725 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:35.831728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:35.831730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:35.831920 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:35.832658 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:35.832665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:35.833041 (+ 376us) client_negotiation.cc:769] Sending connection context
0926 02:19:35.833284 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 02:19:35.833702 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":311,"threads_started":1}
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:19:35.836653 14032 heartbeater.cc:347] Connected to a master server at 127.12.244.254:38115
I20260926 02:19:35.836980 14032 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:35.837756 14032 heartbeater.cc:511] Master 127.12.244.254:38115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:19:35.840260 13843 ts_manager.cc:194] Registered new tserver with Master: 87574fecf2624703b13e3585ac8378c4 (127.12.244.193:42063)
I20260926 02:19:35.842475 13843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:37023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:19:35.862632 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:38115
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:36.198905 14038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:36.199316 14038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:36.199503 14038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:36.207078 14038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:36.207283 14038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:36.207463 14038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:19:36.217661 14038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:38115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:36.219197 14038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:36.220579 14038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:36.234680 14043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:36.234743 14046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:36.234721 14044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:36.239825 14038 server_base.cc:1034] running on GCE node
I20260926 02:19:36.240579 14038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:36.241640 14038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:36.242861 14038 hybrid_clock.cc:648] HybridClock initialized: now 1790389176242782 us; error 66 us; skew 500 ppm
I20260926 02:19:36.243264 14038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:36 dist-test-slave-qfmw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389176, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:36.246532 14038 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:19:36.248325 14038 webserver.cc:460] Webserver started at http://127.12.244.194:34733/ using document root <none> and password file <none>
I20260926 02:19:36.249046 14038 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:36.249150 14038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:36.249418 14038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:36.252604 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "699a2409a1dd474e804fcc0aae87f8d4"
format_stamp: "Formatted at 2026-09-26 02:19:36 on dist-test-slave-qfmw"
server_key: "47b4fe621ee55dc79fb9e576bcc52086"
server_key_iv: "7a9698f2ee829ba2302783b56dce4d1f"
server_key_version: "encryptionkey@0"
I20260926 02:19:36.253404 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "699a2409a1dd474e804fcc0aae87f8d4"
format_stamp: "Formatted at 2026-09-26 02:19:36 on dist-test-slave-qfmw"
server_key: "47b4fe621ee55dc79fb9e576bcc52086"
server_key_iv: "7a9698f2ee829ba2302783b56dce4d1f"
server_key_version: "encryptionkey@0"
I20260926 02:19:36.259063 14038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:19:36.263322 14053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:36.264945 14038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:19:36.265175 14038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "699a2409a1dd474e804fcc0aae87f8d4"
format_stamp: "Formatted at 2026-09-26 02:19:36 on dist-test-slave-qfmw"
server_key: "47b4fe621ee55dc79fb9e576bcc52086"
server_key_iv: "7a9698f2ee829ba2302783b56dce4d1f"
server_key_version: "encryptionkey@0"
I20260926 02:19:36.265394 14038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:36.282511 14038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:36.287635 14038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:36.288004 14038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:36.289093 14038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:36.290715 14038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:36.290795 14038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:36.290980 14038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:36.291041 14038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:36.328483 14038 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:46081
I20260926 02:19:36.328532 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:46081 every 8 connection(s)
I20260926 02:19:36.330018 14038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:19:36.331580 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 14038
I20260926 02:19:36.331791 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:19:36.332518 13267 external_mini_cluster.cc:1428] Setting key 6d9ed44834cf77edb593cf5c96ef0aac
Sep 26 02:19:36 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389176, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
I20260926 02:19:36.355082 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.332948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:42953 (local address 127.12.244.254:38115)
0926 02:19:36.333127 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:36.333132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:36.334926 (+ 1794us) server_negotiation.cc:407] Connection header received
0926 02:19:36.337084 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:36.337092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:36.337187 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:36.337306 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:36.339061 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.339809 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.342476 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.342898 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.343007 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.348160 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:36.348167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:36.348170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:36.348207 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:36.351439 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:36.352337 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:36.352344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:36.352349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:36.352502 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:36.353040 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:36.353044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:36.353046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:36.353398 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:36.353573 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:36.354318 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 02:19:36.354607 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:19:36.355835 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.333794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:38115 (local address 127.12.244.194:42953)
0926 02:19:36.334747 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:36.334797 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:36.336774 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:36.337553 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:36.337563 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:36.338106 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:36.338811 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.338839 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.339975 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.339983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.342308 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.342326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.343174 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.343184 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.343540 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.344499 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:36.344535 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:36.347843 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:36.351646 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:36.351655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:36.351673 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:36.352138 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:36.352625 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:36.352629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:36.352632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:36.352921 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:36.353542 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:36.353550 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:36.353904 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 02:19:36.354157 (+ 253us) client_negotiation.cc:240] Negotiation successful
0926 02:19:36.354582 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":385,"threads_started":1}
I20260926 02:19:36.357458 14166 heartbeater.cc:347] Connected to a master server at 127.12.244.254:38115
I20260926 02:19:36.357831 14166 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:36.358621 14166 heartbeater.cc:511] Master 127.12.244.254:38115 requested a full tablet report, sending...
I20260926 02:19:36.360504 13843 ts_manager.cc:194] Registered new tserver with Master: 699a2409a1dd474e804fcc0aae87f8d4 (127.12.244.194:46081)
I20260926 02:19:36.361946 13843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:42953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:19:36.379056 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:38115
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:36.720508 14171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:36.720818 14171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:36.720991 14171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:36.728518 14171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:36.728643 14171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:36.728824 14171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:19:36.739111 14171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:38115
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:36.740466 14171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:36.741851 14171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:36.755607 14179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:36.755669 14176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:36.755745 14177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:36.757145 14171 server_base.cc:1034] running on GCE node
I20260926 02:19:36.757829 14171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:36.758948 14171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:36.760224 14171 hybrid_clock.cc:648] HybridClock initialized: now 1790389176760187 us; error 76 us; skew 500 ppm
I20260926 02:19:36.760568 14171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:36 dist-test-slave-qfmw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389176, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:36.763936 14171 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:19:36.765623 14171 webserver.cc:460] Webserver started at http://127.12.244.195:37557/ using document root <none> and password file <none>
I20260926 02:19:36.766366 14171 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:36.766525 14171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:36.766852 14171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:36.770043 14171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "5711fcd1248e47ca8b6253084eeacf58"
format_stamp: "Formatted at 2026-09-26 02:19:36 on dist-test-slave-qfmw"
server_key: "fa83e3cb63184fe19484938ff8ca7d14"
server_key_iv: "7d4d4e09591d036e1ce4adb31ac2c6bd"
server_key_version: "encryptionkey@0"
I20260926 02:19:36.770908 14171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "5711fcd1248e47ca8b6253084eeacf58"
format_stamp: "Formatted at 2026-09-26 02:19:36 on dist-test-slave-qfmw"
server_key: "fa83e3cb63184fe19484938ff8ca7d14"
server_key_iv: "7d4d4e09591d036e1ce4adb31ac2c6bd"
server_key_version: "encryptionkey@0"
I20260926 02:19:36.776400 14171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:19:36.780081 14186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:36.781890 14171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:19:36.782097 14171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "5711fcd1248e47ca8b6253084eeacf58"
format_stamp: "Formatted at 2026-09-26 02:19:36 on dist-test-slave-qfmw"
server_key: "fa83e3cb63184fe19484938ff8ca7d14"
server_key_iv: "7d4d4e09591d036e1ce4adb31ac2c6bd"
server_key_version: "encryptionkey@0"
I20260926 02:19:36.782408 14171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:36.799289 14171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:36.803956 14171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:36.804270 14171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:36.805298 14171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:36.806790 14171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:36.806876 14171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:36.807015 14171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:36.807109 14171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:36.842664 14171 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:43975
I20260926 02:19:36.842772 14298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:43975 every 8 connection(s)
I20260926 02:19:36.844125 14171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:19:36.845854 14032 heartbeater.cc:503] Master 127.12.244.254:38115 was elected leader, sending a full tablet report...
I20260926 02:19:36.848035 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 14171
I20260926 02:19:36.848233 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:19:36.848673 13267 external_mini_cluster.cc:1428] Setting key d0a9c9e1493265cbbeaeb9a5d2e0573e
Sep 26 02:19:36 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389176, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:36.867389 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.847332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:51593 (local address 127.12.244.254:38115)
0926 02:19:36.847549 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:36.847555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:36.849279 (+ 1724us) server_negotiation.cc:407] Connection header received
0926 02:19:36.851569 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:36.851577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:36.851667 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:36.851757 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:36.853434 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.854206 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.855985 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.856430 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.856538 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.860814 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:36.860819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:36.860823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:36.860859 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:36.863839 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:36.864918 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:36.864922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:36.864924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:36.865106 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:36.865573 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:36.865576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:36.865578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:36.865840 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:36.865952 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:36.866848 (+ 896us) server_negotiation.cc:299] Negotiation successful
0926 02:19:36.867038 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:19:36.868430 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.848100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:38115 (local address 127.12.244.195:51593)
0926 02:19:36.849087 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:36.849141 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:36.851212 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:36.852048 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:36.852058 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:36.852641 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:36.853260 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.853279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.854376 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.854380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.855846 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.855859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.856684 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.856689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.856919 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.857665 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:36.857690 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:36.860557 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:36.864013 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:36.864052 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:36.864074 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:36.864797 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:36.865260 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:36.865264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:36.865267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:36.865472 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:36.865984 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:36.865992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:36.866362 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 02:19:36.866631 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 02:19:36.867227 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":408,"threads_started":1}
I20260926 02:19:36.870085 14299 heartbeater.cc:347] Connected to a master server at 127.12.244.254:38115
I20260926 02:19:36.870460 14299 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:36.871413 14299 heartbeater.cc:511] Master 127.12.244.254:38115 requested a full tablet report, sending...
I20260926 02:19:36.873167 13843 ts_manager.cc:194] Registered new tserver with Master: 5711fcd1248e47ca8b6253084eeacf58 (127.12.244.195:43975)
I20260926 02:19:36.874001 13843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:51593
I20260926 02:19:36.877756 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:19:36.898774 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.884371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.12.244.254:38115)
0926 02:19:36.884639 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:36.884647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:36.885132 (+ 485us) server_negotiation.cc:407] Connection header received
0926 02:19:36.885363 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:36.885371 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:36.885460 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:36.885570 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:36.887984 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.888760 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.890168 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.890586 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.890735 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.893232 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:36.893238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:36.893241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:36.893281 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:36.895880 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:36.896646 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:36.896652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:36.896656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:36.896803 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:36.897209 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:36.897215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:36.897218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:36.897555 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:36.897811 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:36.898160 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 02:19:36.898386 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:19:36.905982 13843 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:19:36.913851 13843 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:36.959797 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.927879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:43975 (local address 127.0.0.1:48474)
0926 02:19:36.950690 (+ 22811us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:36.950716 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:36.950879 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:36.952130 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:36.952138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:36.952172 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:36.952629 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.952650 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.954781 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.954787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.957034 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.957051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.958647 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.958653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.958881 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.958945 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 02:19:36.959130 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 02:19:36.959352 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22651,"mutex_wait_us":89,"thread_start_us":17822,"threads_started":1}
I20260926 02:19:36.960632 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.928439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.12.244.195:43975)
0926 02:19:36.949878 (+ 21439us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:36.949891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:36.951048 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 02:19:36.951208 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:36.951218 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:36.951558 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:36.951781 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:36.952869 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.954624 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.957222 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.958420 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.958647 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.958846 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:36.960074 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0926 02:19:36.960276 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21174,"thread_start_us":21491,"threads_started":1}
I20260926 02:19:36.961625 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.929242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.12.244.193:42063)
0926 02:19:36.951201 (+ 21959us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:36.951210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:36.953054 (+ 1844us) server_negotiation.cc:407] Connection header received
0926 02:19:36.953242 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:36.953252 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:36.953516 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:36.953744 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:36.954284 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.955766 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.958367 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.959974 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.960176 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.960348 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:36.961079 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 02:19:36.961280 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21741,"thread_start_us":21574,"threads_started":1}
I20260926 02:19:36.961760 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.928457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:42063 (local address 127.0.0.1:60434)
0926 02:19:36.952908 (+ 24451us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:36.952928 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:36.953061 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:36.953725 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:36.953733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:36.953762 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:36.954132 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.954141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.955919 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.955926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.958206 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.958221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.960506 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.960512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.960708 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.960754 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 02:19:36.960875 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 02:19:36.961378 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24301,"thread_start_us":22918,"threads_started":1}
I20260926 02:19:36.961805 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.927507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:46081 (local address 127.0.0.1:53174)
0926 02:19:36.947209 (+ 19702us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:36.947269 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:36.947546 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:36.953341 (+ 5795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:36.953349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:36.953376 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:36.953758 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.953771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.955555 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.955562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.957923 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:36.957946 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.960908 (+ 2962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.960914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:36.961093 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.961123 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:19:36.961215 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:19:36.961348 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19301,"mutex_wait_us":47,"thread_start_us":22856,"threads_started":1}
I20260926 02:19:36.962568 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:36.927736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.12.244.194:46081)
0926 02:19:36.952629 (+ 24893us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:36.952638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:36.952671 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:19:36.952779 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:36.952788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:36.953093 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:36.953344 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:36.953989 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.955382 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.958137 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:36.960384 (+ 2247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:36.960548 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:36.960719 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:36.962006 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0926 02:19:36.962218 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24728,"thread_start_us":24613,"threads_started":1}
I20260926 02:19:36.965389 14234 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd62ca7bf5b4b79be11257a34321f7e (DEFAULT_TABLE table=test-table [id=57cb2b344c8c45b0bb43a4e95af026a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:36.965389 13967 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd62ca7bf5b4b79be11257a34321f7e (DEFAULT_TABLE table=test-table [id=57cb2b344c8c45b0bb43a4e95af026a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:36.965375 14101 tablet_service.cc:1467] Processing CreateTablet for tablet 2bd62ca7bf5b4b79be11257a34321f7e (DEFAULT_TABLE table=test-table [id=57cb2b344c8c45b0bb43a4e95af026a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:36.966820 14101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd62ca7bf5b4b79be11257a34321f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:36.966907 14234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd62ca7bf5b4b79be11257a34321f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:36.966908 13967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bd62ca7bf5b4b79be11257a34321f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:36.978722 14317 tablet_bootstrap.cc:492] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4: Bootstrap starting.
I20260926 02:19:36.983043 14317 tablet_bootstrap.cc:654] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:36.983201 14318 tablet_bootstrap.cc:492] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4: Bootstrap starting.
I20260926 02:19:36.984665 14317 log.cc:824] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:36.986344 14319 tablet_bootstrap.cc:492] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58: Bootstrap starting.
I20260926 02:19:36.987543 14318 tablet_bootstrap.cc:654] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:36.988561 14317 tablet_bootstrap.cc:492] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4: No bootstrap required, opened a new log
I20260926 02:19:36.988871 14317 ts_tablet_manager.cc:1397] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:19:36.988871 14318 log.cc:824] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:36.991072 14319 tablet_bootstrap.cc:654] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:36.992192 14319 log.cc:824] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:36.992229 14318 tablet_bootstrap.cc:492] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4: No bootstrap required, opened a new log
I20260926 02:19:36.992534 14318 ts_tablet_manager.cc:1397] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 02:19:37.000126 14317 raft_consensus.cc:348] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.000653 14317 raft_consensus.cc:374] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:37.000756 14317 raft_consensus.cc:729] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699a2409a1dd474e804fcc0aae87f8d4, State: Initialized, Role: FOLLOWER
I20260926 02:19:37.001552 14319 tablet_bootstrap.cc:492] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58: No bootstrap required, opened a new log
I20260926 02:19:37.001452 14317 consensus_queue.cc:260] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.001886 14319 ts_tablet_manager.cc:1397] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260926 02:19:37.002714 14166 heartbeater.cc:503] Master 127.12.244.254:38115 was elected leader, sending a full tablet report...
I20260926 02:19:37.003720 14318 raft_consensus.cc:348] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.004191 14318 raft_consensus.cc:374] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:37.004257 14317 ts_tablet_manager.cc:1428] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260926 02:19:37.004309 14318 raft_consensus.cc:729] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87574fecf2624703b13e3585ac8378c4, State: Initialized, Role: FOLLOWER
I20260926 02:19:37.004987 14318 consensus_queue.cc:260] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.006955 14318 ts_tablet_manager.cc:1428] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 02:19:37.013639 14319 raft_consensus.cc:348] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.014038 14319 raft_consensus.cc:374] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:37.014107 14319 raft_consensus.cc:729] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5711fcd1248e47ca8b6253084eeacf58, State: Initialized, Role: FOLLOWER
I20260926 02:19:37.014648 14319 consensus_queue.cc:260] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.015756 14299 heartbeater.cc:503] Master 127.12.244.254:38115 was elected leader, sending a full tablet report...
I20260926 02:19:37.016443 14319 ts_tablet_manager.cc:1428] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
W20260926 02:19:37.061672 14033 tablet.cc:2369] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:19:37.082341 14167 tablet.cc:2369] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:37.087580 14324 raft_consensus.cc:482] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:37.087954 14324 raft_consensus.cc:504] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.089776 14324 leader_election.cc:288] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 699a2409a1dd474e804fcc0aae87f8d4 (127.12.244.194:46081), 5711fcd1248e47ca8b6253084eeacf58 (127.12.244.195:43975)
W20260926 02:19:37.096036 14300 tablet.cc:2369] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:37.100394 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.090360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:37543 (local address 127.12.244.195:43975)
0926 02:19:37.090546 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.090553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.091708 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 02:19:37.091913 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.091919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.092001 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.092117 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:37.093057 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.094258 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.097965 (+ 3707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.099385 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.099573 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.099652 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.099980 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.100122 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:19:37.100555 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.090269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:43975 (local address 127.12.244.193:37543)
0926 02:19:37.091548 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:37.091569 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:37.091786 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:37.092162 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:37.092167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:37.092200 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:37.092815 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:37.092829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.094435 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.094441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:37.097724 (+ 3283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:37.097747 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.099628 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.099634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:37.099797 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.099828 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:19:37.099925 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:19:37.100116 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":229,"spinlock_wait_cycles":41088,"thread_start_us":447,"threads_started":1}
I20260926 02:19:37.101359 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.090291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:33441 (local address 127.12.244.194:46081)
0926 02:19:37.090510 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.090517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.091715 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 02:19:37.091913 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.091919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.092059 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.092150 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:37.093080 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.094234 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.099064 (+ 4830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.100295 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.100473 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.100558 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.101032 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.101136 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:19:37.101581 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.090301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:46081 (local address 127.12.244.193:33441)
0926 02:19:37.091607 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:37.091623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:37.091749 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:37.092162 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:37.092167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:37.092207 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:37.092822 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:37.092830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.094441 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.094445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:37.097724 (+ 3279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:37.097747 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.100572 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.100580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:37.100786 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.100821 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:19:37.100988 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 02:19:37.101147 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":419,"spinlock_wait_cycles":4864,"thread_start_us":412,"threads_started":1}
I20260926 02:19:37.101953 14254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd62ca7bf5b4b79be11257a34321f7e" candidate_uuid: "87574fecf2624703b13e3585ac8378c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5711fcd1248e47ca8b6253084eeacf58" is_pre_election: true
I20260926 02:19:37.102519 14254 raft_consensus.cc:2454] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87574fecf2624703b13e3585ac8378c4 in term 0.
I20260926 02:19:37.102696 14121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd62ca7bf5b4b79be11257a34321f7e" candidate_uuid: "87574fecf2624703b13e3585ac8378c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699a2409a1dd474e804fcc0aae87f8d4" is_pre_election: true
I20260926 02:19:37.103298 13922 leader_election.cc:302] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5711fcd1248e47ca8b6253084eeacf58, 87574fecf2624703b13e3585ac8378c4; no voters:
I20260926 02:19:37.103498 14121 raft_consensus.cc:2454] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87574fecf2624703b13e3585ac8378c4 in term 0.
I20260926 02:19:37.103731 14324 raft_consensus.cc:2781] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:37.103897 14324 raft_consensus.cc:482] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:37.104022 14324 raft_consensus.cc:3037] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:37.107216 14324 raft_consensus.cc:504] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.107940 14324 leader_election.cc:288] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [CANDIDATE]: Term 1 election: Requested vote from peers 699a2409a1dd474e804fcc0aae87f8d4 (127.12.244.194:46081), 5711fcd1248e47ca8b6253084eeacf58 (127.12.244.195:43975)
I20260926 02:19:37.108493 14254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd62ca7bf5b4b79be11257a34321f7e" candidate_uuid: "87574fecf2624703b13e3585ac8378c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5711fcd1248e47ca8b6253084eeacf58"
I20260926 02:19:37.108728 14254 raft_consensus.cc:3037] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:37.111183 14121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bd62ca7bf5b4b79be11257a34321f7e" candidate_uuid: "87574fecf2624703b13e3585ac8378c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "699a2409a1dd474e804fcc0aae87f8d4"
I20260926 02:19:37.111279 14254 raft_consensus.cc:2454] T 2bd62ca7bf5b4b79be11257a34321f7e P 5711fcd1248e47ca8b6253084eeacf58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87574fecf2624703b13e3585ac8378c4 in term 1.
I20260926 02:19:37.111438 14121 raft_consensus.cc:3037] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:37.111747 13922 leader_election.cc:302] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5711fcd1248e47ca8b6253084eeacf58, 87574fecf2624703b13e3585ac8378c4; no voters:
I20260926 02:19:37.112089 14324 raft_consensus.cc:2781] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:37.112608 14324 raft_consensus.cc:686] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [term 1 LEADER]: Becoming Leader. State: Replica: 87574fecf2624703b13e3585ac8378c4, State: Running, Role: LEADER
I20260926 02:19:37.113113 14324 consensus_queue.cc:237] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } }
I20260926 02:19:37.114326 14121 raft_consensus.cc:2454] T 2bd62ca7bf5b4b79be11257a34321f7e P 699a2409a1dd474e804fcc0aae87f8d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87574fecf2624703b13e3585ac8378c4 in term 1.
I20260926 02:19:37.118491 13842 catalog_manager.cc:5495] T 2bd62ca7bf5b4b79be11257a34321f7e P 87574fecf2624703b13e3585ac8378c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87574fecf2624703b13e3585ac8378c4 (127.12.244.193). New cstate: current_term: 1 leader_uuid: "87574fecf2624703b13e3585ac8378c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "699a2409a1dd474e804fcc0aae87f8d4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5711fcd1248e47ca8b6253084eeacf58" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87574fecf2624703b13e3585ac8378c4" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42063 } health_report { overall_health: HEALTHY } } }
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:19:37.142225 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.125693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.12.244.193:42063)
0926 02:19:37.125964 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.125971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.126364 (+ 393us) server_negotiation.cc:407] Connection header received
0926 02:19:37.126591 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.126596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.126659 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.126776 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:37.128801 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.129545 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.130869 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.131351 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.131521 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.136179 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:37.136196 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:37.136214 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:37.136257 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:37.139360 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.140131 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.140139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.140141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.140300 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.140685 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.140692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.140695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.141056 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:37.141352 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.141654 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.141887 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:19:37.196107 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.180760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.12.244.193:42063)
0926 02:19:37.181017 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.181026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.181341 (+ 315us) server_negotiation.cc:407] Connection header received
0926 02:19:37.181590 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.181595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.181663 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.181774 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:37.183191 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.183942 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.185259 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.185757 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.185904 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.190115 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:37.190121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:37.190125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:37.190170 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:37.193199 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.193966 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.193971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.193973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.194150 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.194607 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.194610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.194612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.194870 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:37.195186 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.195545 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.195732 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:19:37.244894 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.229855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.12.244.193:42063)
0926 02:19:37.230156 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.230163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.230583 (+ 420us) server_negotiation.cc:407] Connection header received
0926 02:19:37.230714 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.230719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.230800 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.230892 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:37.232407 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.233178 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.234489 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.234892 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.235044 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.239294 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:37.239300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:37.239303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:37.239343 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:37.242042 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.242925 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.242930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.242932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.243184 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.243582 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.243586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.243588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.243918 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:37.244117 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.244413 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.244593 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 02:19:37.246609 13947 tablet_service.cc:2908] Rejecting scan request for tablet 2bd62ca7bf5b4b79be11257a34321f7e: Uninitialized: clean time has not yet been initialized
I20260926 02:19:37.250201 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 13904
I20260926 02:19:37.264505 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.251297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.12.244.193:42063)
0926 02:19:37.251541 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.251547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.251728 (+ 181us) server_negotiation.cc:407] Connection header received
0926 02:19:37.251898 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.251902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.251964 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.252049 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:37.253281 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.254036 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.255304 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.255719 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.255819 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.258077 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:37.258083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:37.258086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:37.258130 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:37.261389 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.262172 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.262177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.262180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.262342 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.262752 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.262756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.262758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.263066 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:37.263404 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.263816 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.264083 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:19:37.265215 14027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:37.359267 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 13904
I20260926 02:19:37.376740 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14038
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:19:37.393175 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.377536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.12.244.194:46081)
0926 02:19:37.377765 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.377771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.377883 (+ 112us) server_negotiation.cc:407] Connection header received
0926 02:19:37.378148 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.378155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.378224 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.378329 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:37.379629 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.380425 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.381725 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.382130 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.382282 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.386933 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:37.386980 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:37.386998 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:37.387043 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:37.390172 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.391060 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.391069 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.391071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.391295 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.391690 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.391693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.391695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.392116 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:37.392363 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.392690 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.392873 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:19:37.393878 14161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:37.483875 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 14038
I20260926 02:19:37.501398 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14171
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:19:37.517872 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:37.502273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.12.244.195:43975)
0926 02:19:37.502451 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:37.502455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:37.502475 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:19:37.502618 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:37.502622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:37.502696 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:37.502784 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:37.504120 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.504922 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.506455 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:37.506882 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:37.507031 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:37.511520 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:37.511537 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:37.511554 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:37.511598 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:37.514708 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.515725 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.515735 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.515738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.515905 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:37.516349 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:37.516353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:37.516355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:37.516740 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:37.517044 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:37.517413 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 02:19:37.517613 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:19:37.518749 14294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:37.613848 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 14171
I20260926 02:19:37.633111 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 13811
I20260926 02:19:37.634461 13873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:37.749444 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 13811
2026-09-26T02:19:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4321 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[14347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[14347](info): set up 2 sockets
Sep 26 02:19:37 dist-test-slave-qfmw krb5kdc[14347](info): commencing operation
krb5kdc: starting...
W20260926 02:19:38.865995 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:19:38 dist-test-slave-qfmw krb5kdc[14347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:19:39.068753 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33475
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:33475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:39.418354 14360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:39.418669 14360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:39.418825 14360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:39.426460 14360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:39.426582 14360 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:19:39.426664 14360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:39.426767 14360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:39.426844 14360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:39.426950 14360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:39.437776 14360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:33475
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:39.439297 14360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:39.440819 14360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:39.453462 14365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:39.453505 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:39.453470 14366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:39.454603 14360 server_base.cc:1034] running on GCE node
I20260926 02:19:39.455603 14360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:39.457268 14360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:39.458513 14360 hybrid_clock.cc:648] HybridClock initialized: now 1790389179458451 us; error 38 us; skew 500 ppm
I20260926 02:19:39.458850 14360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:39 dist-test-slave-qfmw krb5kdc[14347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389179, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:39.462483 14360 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:19:39.464390 14360 webserver.cc:460] Webserver started at http://127.12.244.254:43081/ using document root <none> and password file <none>
I20260926 02:19:39.465183 14360 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:39.465282 14360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:39.465600 14360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:39.468943 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "6cab5f88b2c94a4cb1287edde68db8a3"
format_stamp: "Formatted at 2026-09-26 02:19:39 on dist-test-slave-qfmw"
server_key: "e4ec63bde714ad66dfc3b83a8e456fa7"
server_key_iv: "25c271c76e1e455343f1b2256b69e48b"
server_key_version: "encryptionkey@0"
I20260926 02:19:39.469815 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "6cab5f88b2c94a4cb1287edde68db8a3"
format_stamp: "Formatted at 2026-09-26 02:19:39 on dist-test-slave-qfmw"
server_key: "e4ec63bde714ad66dfc3b83a8e456fa7"
server_key_iv: "25c271c76e1e455343f1b2256b69e48b"
server_key_version: "encryptionkey@0"
I20260926 02:19:39.475914 14360 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 02:19:39.480093 14375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:39.481703 14360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260926 02:19:39.481937 14360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "6cab5f88b2c94a4cb1287edde68db8a3"
format_stamp: "Formatted at 2026-09-26 02:19:39 on dist-test-slave-qfmw"
server_key: "e4ec63bde714ad66dfc3b83a8e456fa7"
server_key_iv: "25c271c76e1e455343f1b2256b69e48b"
server_key_version: "encryptionkey@0"
I20260926 02:19:39.482169 14360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:39.496803 14360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:39.501729 14360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:39.502043 14360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:39.524458 14360 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:33475
I20260926 02:19:39.524472 14436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:33475 every 8 connection(s)
I20260926 02:19:39.525965 14360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:39.528469 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 14360
I20260926 02:19:39.528700 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:19:39.529342 13267 external_mini_cluster.cc:1428] Setting key cec64997cd3e874cf5e99210a46f458d
I20260926 02:19:39.530282 14437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:39.545408 14437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3: Bootstrap starting.
Sep 26 02:19:39 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:39.549870 14437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:39.551589 14437 log.cc:824] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:39.555433 14437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3: No bootstrap required, opened a new log
I20260926 02:19:39.556746 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:39.531494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.12.244.254:33475)
0926 02:19:39.532301 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:39.532320 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:39.532375 (+ 55us) server_negotiation.cc:407] Connection header received
0926 02:19:39.534151 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:39.534186 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:39.534833 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:39.535430 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:39.537074 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:39.539280 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:39.540898 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:39.542029 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:39.542233 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:39.547076 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:39.547111 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:39.547139 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:39.547218 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:39.551627 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:39.552696 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:39.552704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:39.552712 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:39.552889 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:39.553317 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:39.553322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:39.553325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:39.554060 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:39.554341 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:39.554763 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 02:19:39.555269 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":340,"threads_started":1}
I20260926 02:19:39.564299 14437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33475 } }
I20260926 02:19:39.564710 14437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:39.564807 14437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cab5f88b2c94a4cb1287edde68db8a3, State: Initialized, Role: FOLLOWER
I20260926 02:19:39.565349 14437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33475 } }
I20260926 02:19:39.565557 14437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:39.565653 14437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:39.565822 14437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:39.568413 14437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33475 } }
I20260926 02:19:39.568912 14437 leader_election.cc:302] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cab5f88b2c94a4cb1287edde68db8a3; no voters:
I20260926 02:19:39.569528 14437 leader_election.cc:288] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:39.569813 14442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:39.570611 14442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 6cab5f88b2c94a4cb1287edde68db8a3, State: Running, Role: LEADER
I20260926 02:19:39.571323 14442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33475 } }
I20260926 02:19:39.571803 14437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:39.574172 14444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cab5f88b2c94a4cb1287edde68db8a3. Latest consensus state: current_term: 1 leader_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33475 } } }
I20260926 02:19:39.574553 14444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:39.575592 14447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:39.575414 14443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cab5f88b2c94a4cb1287edde68db8a3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33475 } } }
I20260926 02:19:39.575771 14443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:39.581965 14447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:39.592667 14447 catalog_manager.cc:1347] Generated new cluster ID: ded04eb4d4684812ac27c550923c033b
I20260926 02:19:39.592921 14447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:39.612849 14447 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:39.615075 14447 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:39.628307 14447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3: Generated new TSK 0
I20260926 02:19:39.629503 14447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:19:39.681057 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33475
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:40.044082 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:40.044399 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:40.044498 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:40.052217 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:40.052356 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:40.052445 14463 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:19:40.052520 14463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:19:40.063201 14463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:33475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:40.064548 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:40.065933 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:40.081029 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:40.081029 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:40.081055 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:40.081499 14463 server_base.cc:1034] running on GCE node
I20260926 02:19:40.082502 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:40.083621 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:40.084892 14463 hybrid_clock.cc:648] HybridClock initialized: now 1790389180084809 us; error 89 us; skew 500 ppm
I20260926 02:19:40.085282 14463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:40 dist-test-slave-qfmw krb5kdc[14347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389180, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:40.088814 14463 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:19:40.090408 14463 webserver.cc:460] Webserver started at http://127.12.244.193:36103/ using document root <none> and password file <none>
I20260926 02:19:40.091287 14463 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:40.091452 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:40.091866 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:40.095338 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "128727f8957d4944bba00c1e9b4357fd"
format_stamp: "Formatted at 2026-09-26 02:19:40 on dist-test-slave-qfmw"
server_key: "b9662cb7b00f66f296b5bbc5d1073cae"
server_key_iv: "b5b65d3f7a19fdb14d1278f16605ef88"
server_key_version: "encryptionkey@0"
I20260926 02:19:40.096237 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "128727f8957d4944bba00c1e9b4357fd"
format_stamp: "Formatted at 2026-09-26 02:19:40 on dist-test-slave-qfmw"
server_key: "b9662cb7b00f66f296b5bbc5d1073cae"
server_key_iv: "b5b65d3f7a19fdb14d1278f16605ef88"
server_key_version: "encryptionkey@0"
I20260926 02:19:40.101949 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:19:40.106086 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:40.107820 14463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:19:40.108075 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "128727f8957d4944bba00c1e9b4357fd"
format_stamp: "Formatted at 2026-09-26 02:19:40 on dist-test-slave-qfmw"
server_key: "b9662cb7b00f66f296b5bbc5d1073cae"
server_key_iv: "b5b65d3f7a19fdb14d1278f16605ef88"
server_key_version: "encryptionkey@0"
I20260926 02:19:40.108342 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:40.134405 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:40.139480 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:40.139783 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:40.142622 14463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:40.142710 14463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:40.142859 14463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:40.142954 14463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:19:40 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389180, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:40.169570 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.143690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:44241 (local address 127.12.244.254:33475)
0926 02:19:40.144137 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:40.144144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:40.145591 (+ 1447us) server_negotiation.cc:407] Connection header received
0926 02:19:40.148434 (+ 2843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:40.148443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:40.148575 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:40.148725 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:40.151328 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.152645 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.155217 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.155870 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.156023 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.161659 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:40.161664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:40.161668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:40.161708 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:40.164972 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.166146 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.166153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.166156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.166322 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.167205 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.167212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.167218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.167559 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:40.167734 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:40.168947 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0926 02:19:40.169208 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260926 02:19:40.171430 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.144425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33475 (local address 127.12.244.193:44241)
0926 02:19:40.145366 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:40.145425 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:40.147986 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:40.149069 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:40.149088 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:40.149943 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:40.151027 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.151050 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.152870 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.152877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.154981 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.155003 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.156144 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.156152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.156397 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.157250 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:40.157305 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:40.161316 (+ 4011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:40.165175 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.165190 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.165222 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.165992 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.166539 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.166551 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.166558 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.166993 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.167829 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:40.167840 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:40.168475 (+ 635us) client_negotiation.cc:769] Sending connection context
0926 02:19:40.168843 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 02:19:40.169493 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":384,"threads_started":1}
I20260926 02:19:40.186237 14463 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45203
I20260926 02:19:40.186218 14598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45203 every 8 connection(s)
I20260926 02:19:40.187925 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:19:40.190590 14599 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33475
I20260926 02:19:40.190855 14599 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:40.191680 14599 heartbeater.cc:511] Master 127.12.244.254:33475 requested a full tablet report, sending...
I20260926 02:19:40.193486 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 14463
I20260926 02:19:40.193710 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:19:40.194664 13267 external_mini_cluster.cc:1428] Setting key 934c069d9a254cd8bc9f91effb2d1684
I20260926 02:19:40.194731 14392 ts_manager.cc:194] Registered new tserver with Master: 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203)
I20260926 02:19:40.195308 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.179620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:33899 (local address 127.12.244.254:33475)
0926 02:19:40.179862 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:40.179871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:40.180810 (+ 939us) server_negotiation.cc:407] Connection header received
0926 02:19:40.181033 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:40.181037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:40.181105 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:40.181227 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:40.182710 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.183692 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.185564 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.186119 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.186261 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.188647 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:40.188652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:40.188656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:40.188688 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:40.191883 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.192972 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.192977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.192980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.193147 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.193530 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.193537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.193540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.194060 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:40.194221 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:40.194611 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 02:19:40.194854 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:19:40.195358 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.179662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33475 (local address 127.12.244.193:33899)
0926 02:19:40.180655 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:40.180682 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:40.180896 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:40.181264 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:40.181269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:40.181944 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:40.182526 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.182548 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.183930 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.183939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.185392 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.185408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.186569 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.186574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.186707 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.187519 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:40.187537 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:40.188437 (+ 900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:40.192081 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.192086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.192091 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.192739 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.193273 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.193283 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.193285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.193417 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.194184 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:40.194189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:40.194373 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 02:19:40.194517 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 02:19:40.194853 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":598,"threads_started":1}
I20260926 02:19:40.196681 14392 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:44241
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:19:40.240142 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33475
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:40.589418 14603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:40.589732 14603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:40.589830 14603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:40.597518 14603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:40.597683 14603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:40.597775 14603 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:19:40.597857 14603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:19:40.608042 14603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:33475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:40.609385 14603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:40.610764 14603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:40.626024 14608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:40.625960 14609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:40.625988 14611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:40.628930 14603 server_base.cc:1034] running on GCE node
I20260926 02:19:40.629688 14603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:40.630779 14603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:40.632074 14603 hybrid_clock.cc:648] HybridClock initialized: now 1790389180631984 us; error 68 us; skew 500 ppm
I20260926 02:19:40.632409 14603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:40 dist-test-slave-qfmw krb5kdc[14347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389180, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:40.635862 14603 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:19:40.637578 14603 webserver.cc:460] Webserver started at http://127.12.244.194:44263/ using document root <none> and password file <none>
I20260926 02:19:40.638352 14603 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:40.638449 14603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:40.638744 14603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:40.642038 14603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "970821858a4d450980f4013a5e187c64"
format_stamp: "Formatted at 2026-09-26 02:19:40 on dist-test-slave-qfmw"
server_key: "1864c8d4ce7d9c2e5d89c3a74f30921f"
server_key_iv: "0fd716a558858028a650448078966216"
server_key_version: "encryptionkey@0"
I20260926 02:19:40.642849 14603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "970821858a4d450980f4013a5e187c64"
format_stamp: "Formatted at 2026-09-26 02:19:40 on dist-test-slave-qfmw"
server_key: "1864c8d4ce7d9c2e5d89c3a74f30921f"
server_key_iv: "0fd716a558858028a650448078966216"
server_key_version: "encryptionkey@0"
I20260926 02:19:40.648769 14603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 02:19:40.652624 14618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:40.654143 14603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:19:40.654304 14603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "970821858a4d450980f4013a5e187c64"
format_stamp: "Formatted at 2026-09-26 02:19:40 on dist-test-slave-qfmw"
server_key: "1864c8d4ce7d9c2e5d89c3a74f30921f"
server_key_iv: "0fd716a558858028a650448078966216"
server_key_version: "encryptionkey@0"
I20260926 02:19:40.654522 14603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:40.694442 14603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:40.699863 14603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:40.700208 14603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:40.703568 14603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:40.703688 14603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:40.703830 14603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:40.703917 14603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:19:40 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389180, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:40.732251 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.704417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:33167 (local address 127.12.244.254:33475)
0926 02:19:40.704891 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:40.704899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:40.706273 (+ 1374us) server_negotiation.cc:407] Connection header received
0926 02:19:40.709912 (+ 3639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:40.709920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:40.710035 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:40.710140 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:40.713206 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.714465 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.717716 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.718146 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.718295 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.723840 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:40.723851 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:40.723858 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:40.723924 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:40.728168 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.729138 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.729147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.729153 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.729336 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.730016 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.730021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.730031 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.730485 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:40.730681 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:40.731570 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 02:19:40.731842 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":241,"threads_started":1}
I20260926 02:19:40.733420 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.705163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33475 (local address 127.12.244.194:33167)
0926 02:19:40.706070 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:40.706132 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:40.709409 (+ 3277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:40.710540 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:40.710558 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:40.711677 (+ 1119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:40.712875 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.712913 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.714770 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.714779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.717492 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.717521 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.718431 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.718439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.718875 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.720002 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:40.720034 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:40.723468 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:40.728423 (+ 4955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.728432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.728448 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.728967 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.729514 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.729522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.729527 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.729873 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.730644 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:40.730657 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:40.731150 (+ 493us) client_negotiation.cc:769] Sending connection context
0926 02:19:40.731495 (+ 345us) client_negotiation.cc:240] Negotiation successful
0926 02:19:40.732081 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":337,"threads_started":1}
I20260926 02:19:40.758214 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.740799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:58475 (local address 127.12.244.254:33475)
0926 02:19:40.741043 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:40.741053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:40.741714 (+ 661us) server_negotiation.cc:407] Connection header received
0926 02:19:40.741962 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:40.741966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:40.742035 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:40.742130 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:40.743850 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.744831 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.747667 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.748298 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.748457 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.751040 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:40.751051 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:40.751060 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:40.751166 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:40.754868 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.756070 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.756075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.756077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.756202 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:40.756611 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:40.756615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:40.756617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:40.756962 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:40.757159 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:40.757590 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 02:19:40.757852 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:19:40.758359 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:40.740768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33475 (local address 127.12.244.194:58475)
0926 02:19:40.741549 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:40.741576 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:40.741788 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:40.742198 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:40.742206 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:40.742993 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:40.743643 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.743672 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.745095 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.745100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.747423 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:40.747448 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:40.748772 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:40.748782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:40.749015 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:40.749889 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:40.749934 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:40.750734 (+ 800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:40.755204 (+ 4470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.755216 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.755225 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.755855 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.756324 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:40.756329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:40.756332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:40.756471 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:40.757120 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:40.757125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:40.757328 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 02:19:40.757484 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 02:19:40.757867 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":387,"threads_started":1}
I20260926 02:19:40.767992 14603 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:43461
I20260926 02:19:40.768061 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:43461 every 8 connection(s)
I20260926 02:19:40.769407 14603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:19:40.770670 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 14603
I20260926 02:19:40.770886 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:19:40.771572 13267 external_mini_cluster.cc:1428] Setting key 324ee2fee457b60477a3e98d651ab835
I20260926 02:19:40.772074 14739 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33475
I20260926 02:19:40.772436 14739 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:40.773213 14739 heartbeater.cc:511] Master 127.12.244.254:33475 requested a full tablet report, sending...
I20260926 02:19:40.774909 14391 ts_manager.cc:194] Registered new tserver with Master: 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461)
I20260926 02:19:40.775991 14391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:33167
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:19:40.816006 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33475
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:41.158905 14743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:41.159370 14743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:41.159560 14743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:41.167272 14743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:41.167444 14743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:41.167603 14743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:19:41.167752 14743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:19:41.178411 14743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:33475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:41.179934 14743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:41.181396 14743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:41.198071 14748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:41.198081 14751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:41.198264 14749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:41.199404 14743 server_base.cc:1034] running on GCE node
I20260926 02:19:41.200178 14743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:41.200634 14599 heartbeater.cc:503] Master 127.12.244.254:33475 was elected leader, sending a full tablet report...
I20260926 02:19:41.201642 14743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:41.202900 14743 hybrid_clock.cc:648] HybridClock initialized: now 1790389181202841 us; error 44 us; skew 500 ppm
I20260926 02:19:41.203348 14743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:41 dist-test-slave-qfmw krb5kdc[14347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389181, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:41.207228 14743 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:19:41.209194 14743 webserver.cc:460] Webserver started at http://127.12.244.195:46171/ using document root <none> and password file <none>
I20260926 02:19:41.210103 14743 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:41.210276 14743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:41.210678 14743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:41.214306 14743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "f1ad8edc866e4b66a759481586e32d5d"
format_stamp: "Formatted at 2026-09-26 02:19:41 on dist-test-slave-qfmw"
server_key: "57402000c1af4a22208fd21618416f5e"
server_key_iv: "d96dc531ec65e26b59767132ae007add"
server_key_version: "encryptionkey@0"
I20260926 02:19:41.215279 14743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "f1ad8edc866e4b66a759481586e32d5d"
format_stamp: "Formatted at 2026-09-26 02:19:41 on dist-test-slave-qfmw"
server_key: "57402000c1af4a22208fd21618416f5e"
server_key_iv: "d96dc531ec65e26b59767132ae007add"
server_key_version: "encryptionkey@0"
I20260926 02:19:41.221215 14743 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:19:41.225510 14758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:41.227241 14743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:19:41.227455 14743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "f1ad8edc866e4b66a759481586e32d5d"
format_stamp: "Formatted at 2026-09-26 02:19:41 on dist-test-slave-qfmw"
server_key: "57402000c1af4a22208fd21618416f5e"
server_key_iv: "d96dc531ec65e26b59767132ae007add"
server_key_version: "encryptionkey@0"
I20260926 02:19:41.227713 14743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:41.243651 14743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:41.248627 14743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:41.248939 14743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:41.251816 14743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:41.251947 14743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:41.252110 14743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:41.252195 14743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:19:41 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389181, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:41.280531 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.252910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:47045 (local address 127.12.244.254:33475)
0926 02:19:41.253132 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.253138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.254775 (+ 1637us) server_negotiation.cc:407] Connection header received
0926 02:19:41.257253 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.257259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.257374 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.257474 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:41.260573 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.261646 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.264641 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.265418 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.265592 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.271703 (+ 6111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:41.271736 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:41.271741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:41.271795 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:41.276362 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:41.277459 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:41.277466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:41.277470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:41.277617 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:41.278149 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:41.278154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:41.278158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:41.278469 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:41.278664 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.279872 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.280150 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:19:41.283334 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.253532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33475 (local address 127.12.244.195:47045)
0926 02:19:41.254461 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.254524 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.256823 (+ 2299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.258117 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.258135 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.259230 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:41.260341 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.260375 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.261819 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.261825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.264314 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.264341 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.265932 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.265938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.266168 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.267075 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:41.267144 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:41.271276 (+ 4132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:41.276576 (+ 5300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:41.276591 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:41.276618 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:41.277299 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:41.277746 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:41.277751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:41.277755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:41.277995 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:41.278632 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:41.278641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:41.279312 (+ 671us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.280832 (+ 1520us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.281484 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":409,"threads_started":1}
I20260926 02:19:41.295997 14743 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:37693
I20260926 02:19:41.296028 14875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:37693 every 8 connection(s)
I20260926 02:19:41.297488 14743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:19:41.300279 14877 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33475
I20260926 02:19:41.300547 14877 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:41.301579 14877 heartbeater.cc:511] Master 127.12.244.254:33475 requested a full tablet report, sending...
I20260926 02:19:41.303289 14391 ts_manager.cc:194] Registered new tserver with Master: f1ad8edc866e4b66a759481586e32d5d (127.12.244.195:37693)
I20260926 02:19:41.304445 14391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:47045
I20260926 02:19:41.306063 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 14743
I20260926 02:19:41.306272 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:19:41.306697 13267 external_mini_cluster.cc:1428] Setting key 7d6a0a2aeb8560080aa5f83c326b4574
I20260926 02:19:41.310411 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:19:41.314386 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.298162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:38535 (local address 127.12.244.254:33475)
0926 02:19:41.298411 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.298421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.300081 (+ 1660us) server_negotiation.cc:407] Connection header received
0926 02:19:41.300303 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.300311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.300399 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.300502 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:41.301923 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.302923 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.304878 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.305402 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.305514 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.307803 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:41.307811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:41.307816 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:41.307878 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:41.311502 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:41.312299 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:41.312304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:41.312306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:41.312495 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:41.312886 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:41.312910 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:41.312913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:41.313220 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:41.313374 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.313789 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.314022 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:19:41.314452 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.298092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33475 (local address 127.12.244.195:38535)
0926 02:19:41.299982 (+ 1890us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.300003 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.300136 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.300699 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.300704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.301395 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:41.301756 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.301769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.303080 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.303085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.304670 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.304686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.305593 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.305602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.305790 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.306574 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:41.306597 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:41.307621 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:41.311658 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:41.311664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:41.311668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:41.312140 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:41.312617 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:41.312622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:41.312626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:41.312767 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:41.313338 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:41.313343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:41.313550 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.313688 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.314023 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"thread_start_us":1572,"threads_started":1}
Sep 26 02:19:41 dist-test-slave-qfmw krb5kdc[14347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:19:41 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:41.360029 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.345559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.12.244.254:33475)
0926 02:19:41.345729 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.345734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.346050 (+ 316us) server_negotiation.cc:407] Connection header received
0926 02:19:41.346207 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.346211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.346272 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.346378 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:41.348035 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.348826 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.350145 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.350556 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.350660 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.354701 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:41.354707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:41.354711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:41.354752 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:41.357350 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:41.358125 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:41.358129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:41.358132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:41.358265 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:41.358632 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:41.358635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:41.358638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:41.358973 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:41.359232 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.359488 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.359654 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:19:41.364379 14391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:19:41.371265 14391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:41.397797 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.385322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.12.244.195:37693)
0926 02:19:41.386629 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.386638 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.386670 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:19:41.388363 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.388376 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.388695 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.389079 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:41.390082 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.391918 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.394805 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.396435 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.396965 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.397185 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.397261 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.397432 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":975,"threads_started":1}
I20260926 02:19:41.397799 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.384920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:37693 (local address 127.0.0.1:50770)
0926 02:19:41.385769 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.385829 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.386056 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.389414 (+ 3358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.389421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.389457 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:41.389804 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.389813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.392585 (+ 2772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.392592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.394628 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.394640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.396673 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.396681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.396894 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.397008 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.397139 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.397357 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":1,"thread_start_us":372,"threads_started":1}
I20260926 02:19:41.400476 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.387903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.12.244.193:45203)
0926 02:19:41.391929 (+ 4026us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.391935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.391957 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:19:41.392078 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.392091 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.392350 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.392599 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:41.393588 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.395077 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.397578 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.399378 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.399567 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.399771 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.400025 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.400200 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3884,"thread_start_us":3757,"threads_started":1}
I20260926 02:19:41.400542 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.387318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45203 (local address 127.0.0.1:47542)
0926 02:19:41.387877 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.387895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.388049 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.392817 (+ 4768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.392827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.392862 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:41.393353 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.393378 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.395243 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.395251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.397376 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.397390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.399601 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.399607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.399801 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.399839 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.399960 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.400115 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":1,"thread_start_us":337,"threads_started":1}
I20260926 02:19:41.400763 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.386597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:43461 (local address 127.0.0.1:39040)
0926 02:19:41.387301 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.387326 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.387489 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.392163 (+ 4674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.392168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.392197 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:41.392795 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.392808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.394762 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.394768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.396449 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.396462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.399898 (+ 3436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.399903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.400092 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.400124 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.400188 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.400321 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":30,"thread_start_us":369,"threads_started":1}
I20260926 02:19:41.401646 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.386873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.12.244.194:43461)
0926 02:19:41.391481 (+ 4608us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.391491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.391529 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:19:41.391624 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.391631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.391912 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.392164 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:41.393033 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.394606 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.398233 (+ 3627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.399640 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.401054 (+ 1414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.401242 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.401300 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.401425 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4431,"thread_start_us":4306,"threads_started":1}
I20260926 02:19:41.402308 14808 tablet_service.cc:1467] Processing CreateTablet for tablet 37fb574f0d9242b4be4154cd674dbb73 (DEFAULT_TABLE table=test-table [id=baae1ff80be240cc85ef8e30708f6127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:41.403916 14808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fb574f0d9242b4be4154cd674dbb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:41.405004 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 37fb574f0d9242b4be4154cd674dbb73 (DEFAULT_TABLE table=test-table [id=baae1ff80be240cc85ef8e30708f6127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:41.405005 14669 tablet_service.cc:1467] Processing CreateTablet for tablet 37fb574f0d9242b4be4154cd674dbb73 (DEFAULT_TABLE table=test-table [id=baae1ff80be240cc85ef8e30708f6127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:41.406535 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fb574f0d9242b4be4154cd674dbb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:41.406561 14669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37fb574f0d9242b4be4154cd674dbb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:41.417588 14897 tablet_bootstrap.cc:492] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Bootstrap starting.
I20260926 02:19:41.420683 14897 tablet_bootstrap.cc:654] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:41.422056 14899 tablet_bootstrap.cc:492] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Bootstrap starting.
I20260926 02:19:41.422711 14897 log.cc:824] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:41.422878 14898 tablet_bootstrap.cc:492] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Bootstrap starting.
I20260926 02:19:41.426841 14898 tablet_bootstrap.cc:654] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:41.427692 14897 tablet_bootstrap.cc:492] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: No bootstrap required, opened a new log
I20260926 02:19:41.427991 14897 ts_tablet_manager.cc:1397] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 02:19:41.428665 14898 log.cc:824] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:41.429394 14899 tablet_bootstrap.cc:654] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:41.430944 14899 log.cc:824] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:41.433820 14898 tablet_bootstrap.cc:492] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: No bootstrap required, opened a new log
I20260926 02:19:41.434109 14898 ts_tablet_manager.cc:1397] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260926 02:19:41.434278 14899 tablet_bootstrap.cc:492] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: No bootstrap required, opened a new log
I20260926 02:19:41.434532 14899 ts_tablet_manager.cc:1397] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260926 02:19:41.437906 14897 raft_consensus.cc:348] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.438400 14897 raft_consensus.cc:374] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:41.438550 14897 raft_consensus.cc:729] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ad8edc866e4b66a759481586e32d5d, State: Initialized, Role: FOLLOWER
I20260926 02:19:41.439303 14897 consensus_queue.cc:260] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.440528 14877 heartbeater.cc:503] Master 127.12.244.254:33475 was elected leader, sending a full tablet report...
I20260926 02:19:41.441404 14897 ts_tablet_manager.cc:1428] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260926 02:19:41.443691 14898 raft_consensus.cc:348] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.444192 14898 raft_consensus.cc:374] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:41.444289 14898 raft_consensus.cc:729] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970821858a4d450980f4013a5e187c64, State: Initialized, Role: FOLLOWER
I20260926 02:19:41.445153 14898 consensus_queue.cc:260] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.445267 14899 raft_consensus.cc:348] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.445714 14899 raft_consensus.cc:374] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:41.445874 14899 raft_consensus.cc:729] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128727f8957d4944bba00c1e9b4357fd, State: Initialized, Role: FOLLOWER
I20260926 02:19:41.446849 14899 consensus_queue.cc:260] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.447199 14898 ts_tablet_manager.cc:1428] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260926 02:19:41.447781 14739 heartbeater.cc:503] Master 127.12.244.254:33475 was elected leader, sending a full tablet report...
I20260926 02:19:41.448575 14899 ts_tablet_manager.cc:1428] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260926 02:19:41.498632 14903 raft_consensus.cc:482] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:41.499001 14903 raft_consensus.cc:504] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.501011 14903 leader_election.cc:288] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461), 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203)
I20260926 02:19:41.502211 14905 raft_consensus.cc:482] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:41.502693 14905 raft_consensus.cc:504] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.508948 14905 leader_election.cc:288] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1ad8edc866e4b66a759481586e32d5d (127.12.244.195:37693), 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461)
I20260926 02:19:41.510211 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.501364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45203 (local address 127.12.244.195:51453)
0926 02:19:41.501707 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.501734 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.501898 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.502349 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.502358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.502392 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:41.502618 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.502630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.504148 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.504154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.506865 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.506889 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.509341 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.509348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.509576 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.509615 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.509718 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.509883 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":48}
I20260926 02:19:41.510993 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.501515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:51453 (local address 127.12.244.193:45203)
0926 02:19:41.501732 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.501741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.501980 (+ 239us) server_negotiation.cc:407] Connection header received
0926 02:19:41.502077 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.502085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.502175 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.502311 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:41.502777 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.503991 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.507054 (+ 3063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.508970 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.509152 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.509235 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.510562 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.510726 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:19:41.511605 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.501415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:43461 (local address 127.12.244.195:44077)
0926 02:19:41.502378 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.502398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.502552 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.502947 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.502952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.502980 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:41.503372 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.503387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.504658 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.504668 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.508641 (+ 3973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.508664 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.510675 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.510682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.510869 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.510906 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.510998 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.511176 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":961,"thread_start_us":603,"threads_started":1}
I20260926 02:19:41.512352 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.501595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:44077 (local address 127.12.244.194:43461)
0926 02:19:41.501785 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.501791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.502497 (+ 706us) server_negotiation.cc:407] Connection header received
0926 02:19:41.502717 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.502724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.502805 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.502911 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:41.503550 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.504511 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.508852 (+ 4341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.510280 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.511750 (+ 1470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.511841 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.511919 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.512071 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:19:41.512111 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fb574f0d9242b4be4154cd674dbb73" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128727f8957d4944bba00c1e9b4357fd" is_pre_election: true
I20260926 02:19:41.512676 14549 raft_consensus.cc:2454] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 0.
I20260926 02:19:41.513624 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fb574f0d9242b4be4154cd674dbb73" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970821858a4d450980f4013a5e187c64" is_pre_election: true
I20260926 02:19:41.514629 14761 leader_election.cc:302] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128727f8957d4944bba00c1e9b4357fd, f1ad8edc866e4b66a759481586e32d5d; no voters:
I20260926 02:19:41.514757 14694 raft_consensus.cc:2454] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 0.
I20260926 02:19:41.515152 14903 raft_consensus.cc:2781] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:41.515298 14903 raft_consensus.cc:482] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:41.515556 14903 raft_consensus.cc:3037] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:41.518853 14903 raft_consensus.cc:504] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.519829 14903 leader_election.cc:288] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 election: Requested vote from peers 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461), 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203)
I20260926 02:19:41.520246 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fb574f0d9242b4be4154cd674dbb73" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970821858a4d450980f4013a5e187c64"
I20260926 02:19:41.520380 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fb574f0d9242b4be4154cd674dbb73" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128727f8957d4944bba00c1e9b4357fd"
I20260926 02:19:41.520479 14694 raft_consensus.cc:3037] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:41.520615 14549 raft_consensus.cc:3037] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:41.521199 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.505467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:42727 (local address 127.12.244.195:37693)
0926 02:19:41.505666 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.505673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.506871 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 02:19:41.512374 (+ 5503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.512382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.512475 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.512561 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:41.514345 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.515605 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.518467 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.520006 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.520170 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.520244 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.520784 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.520956 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:19:41.521241 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.505222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:43461 (local address 127.12.244.193:34481)
0926 02:19:41.509715 (+ 4493us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.509740 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.512231 (+ 2491us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x612000022d28. stack: 00007f1daf76a66e 00007f1daf76a430 00007f1dae7f15ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f1daf7600fd 00007f1daf75fd1e 00007f1dc26693cd 00007f1dc266936c 00007f1dc2635b3b 00007f1db616f536 00007f1db616dd9e 00007f1db61f5dca 00007f1db61f3f68 00007f1db622782f
0926 02:19:41.512271 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.512734 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.512739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.512775 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:41.513252 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.513265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.515883 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.515897 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.517922 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.517943 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.520135 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.520141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.520351 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.520408 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.520517 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.520681 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4296,"mutex_wait_us":100,"spinlock_wait_cycles":3246464,"thread_start_us":1290,"threads_started":1}
W20260926 02:19:41.521651 14740 tablet.cc:2369] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:41.522069 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.505062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:37693 (local address 127.12.244.193:42727)
0926 02:19:41.506739 (+ 1677us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:41.506767 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:41.512176 (+ 5409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:41.512768 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:41.512778 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:41.512810 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:41.514169 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.514183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.515824 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.515829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.518286 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:41.518301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.520417 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.520423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:41.520605 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.520637 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:19:41.521383 (+ 746us) client_negotiation.cc:240] Negotiation successful
0926 02:19:41.521545 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1462,"mutex_wait_us":66,"thread_start_us":5726,"threads_started":1}
I20260926 02:19:41.522943 14828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fb574f0d9242b4be4154cd674dbb73" candidate_uuid: "128727f8957d4944bba00c1e9b4357fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ad8edc866e4b66a759481586e32d5d" is_pre_election: true
I20260926 02:19:41.523821 14549 raft_consensus.cc:2454] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 1.
I20260926 02:19:41.523993 14828 raft_consensus.cc:2379] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 128727f8957d4944bba00c1e9b4357fd in current term 1: Already voted for candidate f1ad8edc866e4b66a759481586e32d5d in this term.
I20260926 02:19:41.524241 14761 leader_election.cc:302] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128727f8957d4944bba00c1e9b4357fd, f1ad8edc866e4b66a759481586e32d5d; no voters:
I20260926 02:19:41.524591 14903 raft_consensus.cc:2781] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:41.525537 14903 raft_consensus.cc:686] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 1 LEADER]: Becoming Leader. State: Replica: f1ad8edc866e4b66a759481586e32d5d, State: Running, Role: LEADER
I20260926 02:19:41.525854 14694 raft_consensus.cc:2454] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 1.
I20260926 02:19:41.526127 14903 consensus_queue.cc:237] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } }
I20260926 02:19:41.526399 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.506262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:34481 (local address 127.12.244.194:43461)
0926 02:19:41.510748 (+ 4486us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.510754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.510782 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:19:41.512473 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.512478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.512573 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.513335 (+ 762us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:41.513702 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.515634 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.518114 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.519720 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.519930 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.520016 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.521452 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.521603 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4355,"thread_start_us":4098,"threads_started":1}
I20260926 02:19:41.527761 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37fb574f0d9242b4be4154cd674dbb73" candidate_uuid: "128727f8957d4944bba00c1e9b4357fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970821858a4d450980f4013a5e187c64" is_pre_election: true
I20260926 02:19:41.528083 14694 raft_consensus.cc:2379] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 128727f8957d4944bba00c1e9b4357fd in current term 1: Already voted for candidate f1ad8edc866e4b66a759481586e32d5d in this term.
I20260926 02:19:41.528903 14479 leader_election.cc:302] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 128727f8957d4944bba00c1e9b4357fd; no voters: 970821858a4d450980f4013a5e187c64, f1ad8edc866e4b66a759481586e32d5d
I20260926 02:19:41.529452 14905 raft_consensus.cc:2726] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:19:41.536254 14391 catalog_manager.cc:5495] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d reported cstate change: term changed from 0 to 1, leader changed from <none> to f1ad8edc866e4b66a759481586e32d5d (127.12.244.195). New cstate: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } health_report { overall_health: UNKNOWN } } }
W20260926 02:19:41.549762 14879 tablet.cc:2369] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:41.564101 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:41.557683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.12.244.195:37693)
0926 02:19:41.557915 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:41.557920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:41.557940 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:19:41.558236 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:41.558240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:41.558319 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:41.558449 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:41.558961 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.559877 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.561533 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:41.561943 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:41.562046 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:41.562625 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:41.563343 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:41.563464 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:41.563634 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 02:19:41.563707 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:19:41.573112 14694 raft_consensus.cc:1261] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Refusing update from remote peer f1ad8edc866e4b66a759481586e32d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:41.573163 14549 raft_consensus.cc:1261] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Refusing update from remote peer f1ad8edc866e4b66a759481586e32d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:41.574054 14903 consensus_queue.cc:1035] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:41.574450 14910 consensus_queue.cc:1035] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:41.587316 14913 mvcc.cc:205] Tried to move back new op lower bound from 7333434087712694272 to 7333434087537512448. Current Snapshot: MvccSnapshot[applied={T|T < 7333434087712694272}]
I20260926 02:19:41.592965 14915 mvcc.cc:205] Tried to move back new op lower bound from 7333434087712694272 to 7333434087537512448. Current Snapshot: MvccSnapshot[applied={T|T < 7333434087712694272}]
I20260926 02:19:41.611045 14392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:41.611378 14392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:41.615741 14392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3: Sending DeleteTablet for 3 replicas of tablet 37fb574f0d9242b4be4154cd674dbb73
I20260926 02:19:41.617175 14808 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fb574f0d9242b4be4154cd674dbb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:41 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:50770
I20260926 02:19:41.617192 14669 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fb574f0d9242b4be4154cd674dbb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:41 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:39040
I20260926 02:19:41.618196 14922 tablet_replica.cc:330] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: stopping tablet replica
I20260926 02:19:41.618216 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet 37fb574f0d9242b4be4154cd674dbb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:41 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:47542
I20260926 02:19:41.618876 14922 raft_consensus.cc:2229] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:19:41.619769 14922 raft_consensus.cc:2258] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:41.621109 14392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:19:41.622015 14392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:41.630266 14924 tablet_replica.cc:330] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: stopping tablet replica
I20260926 02:19:41.630846 14924 raft_consensus.cc:2229] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:41.631403 14924 raft_consensus.cc:2258] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:41.633761 14808 tablet_service.cc:1467] Processing CreateTablet for tablet bed05bdc53534005b04507ea02bbfad1 (DEFAULT_TABLE table=test-table [id=17aa8beb0d144cb386797efbd90e9a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:41.633942 14924 ts_tablet_manager.cc:1905] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:41.634300 14808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed05bdc53534005b04507ea02bbfad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:41.637161 14923 tablet_replica.cc:330] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: stopping tablet replica
I20260926 02:19:41.639508 14922 ts_tablet_manager.cc:1905] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:41.639787 14529 tablet_service.cc:1467] Processing CreateTablet for tablet bed05bdc53534005b04507ea02bbfad1 (DEFAULT_TABLE table=test-table [id=17aa8beb0d144cb386797efbd90e9a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:41.640312 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed05bdc53534005b04507ea02bbfad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:41.642156 14897 tablet_bootstrap.cc:492] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: Bootstrap starting.
I20260926 02:19:41.644040 14897 tablet_bootstrap.cc:654] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:41.648900 14922 ts_tablet_manager.cc:1918] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:19:41.649143 14922 log.cc:1196] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/37fb574f0d9242b4be4154cd674dbb73
I20260926 02:19:41.649776 14922 ts_tablet_manager.cc:1939] T 37fb574f0d9242b4be4154cd674dbb73 P f1ad8edc866e4b66a759481586e32d5d: Deleting consensus metadata
I20260926 02:19:41.652160 14378 catalog_manager.cc:4841] TS f1ad8edc866e4b66a759481586e32d5d (127.12.244.195:37693): tablet 37fb574f0d9242b4be4154cd674dbb73 (table test-table [id=baae1ff80be240cc85ef8e30708f6127]) successfully deleted
I20260926 02:19:41.640057 14669 tablet_service.cc:1467] Processing CreateTablet for tablet bed05bdc53534005b04507ea02bbfad1 (DEFAULT_TABLE table=test-table [id=17aa8beb0d144cb386797efbd90e9a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:41.653829 14669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bed05bdc53534005b04507ea02bbfad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:41.654114 14897 tablet_bootstrap.cc:492] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: No bootstrap required, opened a new log
I20260926 02:19:41.654307 14897 ts_tablet_manager.cc:1397] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260926 02:19:41.655292 14897 raft_consensus.cc:348] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:41.655640 14897 raft_consensus.cc:374] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:41.655795 14897 raft_consensus.cc:729] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ad8edc866e4b66a759481586e32d5d, State: Initialized, Role: FOLLOWER
I20260926 02:19:41.637653 14923 raft_consensus.cc:2229] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:41.656385 14897 consensus_queue.cc:260] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:41.656577 14923 raft_consensus.cc:2258] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:41.658123 14924 ts_tablet_manager.cc:1918] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:19:41.658406 14924 log.cc:1196] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/37fb574f0d9242b4be4154cd674dbb73
I20260926 02:19:41.658465 14897 ts_tablet_manager.cc:1428] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 02:19:41.659034 14924 ts_tablet_manager.cc:1939] T 37fb574f0d9242b4be4154cd674dbb73 P 128727f8957d4944bba00c1e9b4357fd: Deleting consensus metadata
I20260926 02:19:41.660971 14378 catalog_manager.cc:4841] TS 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203): tablet 37fb574f0d9242b4be4154cd674dbb73 (table test-table [id=baae1ff80be240cc85ef8e30708f6127]) successfully deleted
I20260926 02:19:41.662456 14923 ts_tablet_manager.cc:1905] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:41.663130 14898 tablet_bootstrap.cc:492] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: Bootstrap starting.
I20260926 02:19:41.664147 14899 tablet_bootstrap.cc:492] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Bootstrap starting.
I20260926 02:19:41.664971 14898 tablet_bootstrap.cc:654] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:41.665729 14899 tablet_bootstrap.cc:654] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:41.672204 14923 ts_tablet_manager.cc:1918] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:19:41.672516 14923 log.cc:1196] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/37fb574f0d9242b4be4154cd674dbb73
I20260926 02:19:41.673244 14923 ts_tablet_manager.cc:1939] T 37fb574f0d9242b4be4154cd674dbb73 P 970821858a4d450980f4013a5e187c64: Deleting consensus metadata
I20260926 02:19:41.674620 14903 raft_consensus.cc:482] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:41.674810 14903 raft_consensus.cc:504] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:41.675561 14903 leader_election.cc:288] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203), 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461)
I20260926 02:19:41.675910 14376 catalog_manager.cc:4841] TS 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461): tablet 37fb574f0d9242b4be4154cd674dbb73 (table test-table [id=baae1ff80be240cc85ef8e30708f6127]) successfully deleted
I20260926 02:19:41.676266 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed05bdc53534005b04507ea02bbfad1" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970821858a4d450980f4013a5e187c64" is_pre_election: true
I20260926 02:19:41.676704 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed05bdc53534005b04507ea02bbfad1" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128727f8957d4944bba00c1e9b4357fd" is_pre_election: true
W20260926 02:19:41.677292 14759 leader_election.cc:341] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461): Illegal state: must be running to vote when last-logged opid is not known
W20260926 02:19:41.677557 14761 leader_election.cc:341] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203): Illegal state: must be running to vote when last-logged opid is not known
I20260926 02:19:41.677699 14761 leader_election.cc:302] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1ad8edc866e4b66a759481586e32d5d; no voters: 128727f8957d4944bba00c1e9b4357fd, 970821858a4d450980f4013a5e187c64
I20260926 02:19:41.678017 14903 raft_consensus.cc:2726] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:19:41.679415 14898 tablet_bootstrap.cc:492] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: No bootstrap required, opened a new log
I20260926 02:19:41.679456 14899 tablet_bootstrap.cc:492] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: No bootstrap required, opened a new log
I20260926 02:19:41.679584 14898 ts_tablet_manager.cc:1397] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260926 02:19:41.679617 14899 ts_tablet_manager.cc:1397] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260926 02:19:41.680902 14898 raft_consensus.cc:348] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:41.681128 14898 raft_consensus.cc:374] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:41.681216 14898 raft_consensus.cc:729] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970821858a4d450980f4013a5e187c64, State: Initialized, Role: FOLLOWER
I20260926 02:19:41.681107 14899 raft_consensus.cc:348] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:41.681326 14899 raft_consensus.cc:374] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:41.681424 14899 raft_consensus.cc:729] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128727f8957d4944bba00c1e9b4357fd, State: Initialized, Role: FOLLOWER
I20260926 02:19:41.681496 14898 consensus_queue.cc:260] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:41.681705 14899 consensus_queue.cc:260] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:41.682678 14898 ts_tablet_manager.cc:1428] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:41.684221 14899 ts_tablet_manager.cc:1428] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260926 02:19:41.690685 14600 tablet.cc:2369] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:42.006239 14904 raft_consensus.cc:482] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:42.006557 14904 raft_consensus.cc:504] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.008284 14904 leader_election.cc:288] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203), f1ad8edc866e4b66a759481586e32d5d (127.12.244.195:37693)
I20260926 02:19:42.017705 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.008762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:55703 (local address 127.12.244.193:45203)
0926 02:19:42.009206 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.009213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.009369 (+ 156us) server_negotiation.cc:407] Connection header received
0926 02:19:42.009685 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.009691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.009781 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.009877 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:42.010674 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.011874 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.015495 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.016478 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.016640 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.016720 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.017204 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.017350 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260926 02:19:42.017988 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.009533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:45343 (local address 127.12.244.195:37693)
0926 02:19:42.009725 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.009730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.010303 (+ 573us) server_negotiation.cc:407] Connection header received
0926 02:19:42.010488 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.010494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.010571 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.010678 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:42.011343 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.012410 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.015438 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.016739 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.016905 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.016985 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.017566 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.017718 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:19:42.018015 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.009394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:37693 (local address 127.12.244.194:45343)
0926 02:19:42.010190 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:42.010211 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:42.010335 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:42.010715 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:42.010721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:42.010750 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:42.011130 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.011143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.012581 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.012586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.015250 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.015268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.017146 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.017153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.017337 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.017369 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:19:42.017514 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:19:42.017654 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":3,"thread_start_us":462,"threads_started":1}
I20260926 02:19:42.018731 14928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.008618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45203 (local address 127.12.244.194:55703)
0926 02:19:42.009217 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:42.009238 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:42.009503 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:42.010112 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:42.010115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:42.010140 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:42.010532 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.010541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.012035 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.012040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.015250 (+ 3210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.015268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.016816 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.016822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.016992 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.017024 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:19:42.018199 (+ 1175us) client_negotiation.cc:240] Negotiation successful
0926 02:19:42.018340 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":97,"thread_start_us":322,"threads_started":1}
I20260926 02:19:42.018699 14828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed05bdc53534005b04507ea02bbfad1" candidate_uuid: "970821858a4d450980f4013a5e187c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ad8edc866e4b66a759481586e32d5d" is_pre_election: true
I20260926 02:19:42.018987 14828 raft_consensus.cc:2454] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 970821858a4d450980f4013a5e187c64 in term 0.
I20260926 02:19:42.019403 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed05bdc53534005b04507ea02bbfad1" candidate_uuid: "970821858a4d450980f4013a5e187c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128727f8957d4944bba00c1e9b4357fd" is_pre_election: true
I20260926 02:19:42.019541 14621 leader_election.cc:302] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 970821858a4d450980f4013a5e187c64, f1ad8edc866e4b66a759481586e32d5d; no voters:
I20260926 02:19:42.019661 14549 raft_consensus.cc:2454] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 970821858a4d450980f4013a5e187c64 in term 0.
I20260926 02:19:42.020080 14904 raft_consensus.cc:2781] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:42.020215 14904 raft_consensus.cc:482] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:42.020328 14904 raft_consensus.cc:3037] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:42.022682 14904 raft_consensus.cc:504] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.023432 14904 leader_election.cc:288] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [CANDIDATE]: Term 1 election: Requested vote from peers 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203), f1ad8edc866e4b66a759481586e32d5d (127.12.244.195:37693)
I20260926 02:19:42.023990 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed05bdc53534005b04507ea02bbfad1" candidate_uuid: "970821858a4d450980f4013a5e187c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128727f8957d4944bba00c1e9b4357fd"
I20260926 02:19:42.024134 14828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bed05bdc53534005b04507ea02bbfad1" candidate_uuid: "970821858a4d450980f4013a5e187c64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ad8edc866e4b66a759481586e32d5d"
I20260926 02:19:42.024214 14549 raft_consensus.cc:3037] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:42.024353 14828 raft_consensus.cc:3037] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:42.027328 14549 raft_consensus.cc:2454] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 970821858a4d450980f4013a5e187c64 in term 1.
I20260926 02:19:42.027639 14828 raft_consensus.cc:2454] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 970821858a4d450980f4013a5e187c64 in term 1.
I20260926 02:19:42.027945 14621 leader_election.cc:302] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128727f8957d4944bba00c1e9b4357fd, 970821858a4d450980f4013a5e187c64; no voters:
I20260926 02:19:42.028291 14904 raft_consensus.cc:2781] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:42.028784 14904 raft_consensus.cc:686] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 1 LEADER]: Becoming Leader. State: Replica: 970821858a4d450980f4013a5e187c64, State: Running, Role: LEADER
I20260926 02:19:42.029191 14904 consensus_queue.cc:237] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.033473 14392 catalog_manager.cc:5495] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 970821858a4d450980f4013a5e187c64 (127.12.244.194). New cstate: current_term: 1 leader_uuid: "970821858a4d450980f4013a5e187c64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } health_report { overall_health: UNKNOWN } } }
Sep 26 02:19:42 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389179, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:42.145145 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.129849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33475 (local address 127.0.0.1:57078)
0926 02:19:42.130733 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:42.130755 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:42.130926 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:42.131483 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:42.131487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:42.132164 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:42.132366 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.132377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.133655 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.133659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.134640 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.134652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.135555 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.135561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.135702 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.136478 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:42.136500 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:42.138918 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:42.142339 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:42.142352 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:42.142359 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:42.142903 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:42.143439 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:42.143443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:42.143446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:42.143588 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:42.144236 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:42.144243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:42.144409 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 02:19:42.144555 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 02:19:42.144784 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":494,"threads_started":1}
I20260926 02:19:42.145404 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.129864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.12.244.254:33475)
0926 02:19:42.130766 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.130770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.130846 (+ 76us) server_negotiation.cc:407] Connection header received
0926 02:19:42.131222 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.131226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.131285 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.131379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:42.132529 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.133522 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.134794 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.135361 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.135470 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.139215 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:42.139221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:42.139224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:42.139257 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:42.142126 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.143031 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.143036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.143039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.143240 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.143727 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.143731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.143733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.144075 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:42.144200 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.144643 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.145024 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":791,"thread_start_us":515,"threads_started":1}
I20260926 02:19:42.151991 14392 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:57078:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:19:42.153174 14392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:42.160523 14669 tablet_service.cc:1467] Processing CreateTablet for tablet 5393cd7732ae49c2988f8a107df89c7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca9c757cb6444ec3af8bb5413aa0b16d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:19:42.160523 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 5393cd7732ae49c2988f8a107df89c7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca9c757cb6444ec3af8bb5413aa0b16d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:19:42.160862 14808 tablet_service.cc:1467] Processing CreateTablet for tablet 5393cd7732ae49c2988f8a107df89c7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca9c757cb6444ec3af8bb5413aa0b16d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:19:42.161139 14669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5393cd7732ae49c2988f8a107df89c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:42.161129 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5393cd7732ae49c2988f8a107df89c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:42.161425 14808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5393cd7732ae49c2988f8a107df89c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:42.168520 14898 tablet_bootstrap.cc:492] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: Bootstrap starting.
I20260926 02:19:42.169167 14939 tablet_bootstrap.cc:492] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: Bootstrap starting.
I20260926 02:19:42.170249 14898 tablet_bootstrap.cc:654] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:42.170769 14939 tablet_bootstrap.cc:654] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:42.171653 14899 tablet_bootstrap.cc:492] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: Bootstrap starting.
I20260926 02:19:42.173333 14939 tablet_bootstrap.cc:492] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: No bootstrap required, opened a new log
I20260926 02:19:42.173355 14899 tablet_bootstrap.cc:654] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:42.173511 14939 ts_tablet_manager.cc:1397] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:19:42.174333 14939 raft_consensus.cc:348] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.174600 14939 raft_consensus.cc:374] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:42.174705 14939 raft_consensus.cc:729] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ad8edc866e4b66a759481586e32d5d, State: Initialized, Role: FOLLOWER
I20260926 02:19:42.174950 14939 consensus_queue.cc:260] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.175350 14898 tablet_bootstrap.cc:492] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: No bootstrap required, opened a new log
I20260926 02:19:42.175514 14939 ts_tablet_manager.cc:1428] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:19:42.175529 14898 ts_tablet_manager.cc:1397] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260926 02:19:42.175408 14903 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.175724 14903 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:42.176415 14898 raft_consensus.cc:348] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.176582 14898 raft_consensus.cc:374] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:42.176676 14898 raft_consensus.cc:729] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970821858a4d450980f4013a5e187c64, State: Initialized, Role: FOLLOWER
I20260926 02:19:42.176928 14898 consensus_queue.cc:260] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.177452 14898 ts_tablet_manager.cc:1428] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 02:19:42.177304 14904 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.177613 14904 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:42.178829 14899 tablet_bootstrap.cc:492] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: No bootstrap required, opened a new log
I20260926 02:19:42.179042 14899 ts_tablet_manager.cc:1397] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 02:19:42.179957 14899 raft_consensus.cc:348] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.180205 14899 raft_consensus.cc:374] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:42.180323 14899 raft_consensus.cc:729] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128727f8957d4944bba00c1e9b4357fd, State: Initialized, Role: FOLLOWER
I20260926 02:19:42.180570 14899 consensus_queue.cc:260] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.181146 14899 ts_tablet_manager.cc:1428] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 02:19:42.180989 14905 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.181284 14905 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:42.299572 14903 raft_consensus.cc:482] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:42.299817 14903 raft_consensus.cc:504] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.300514 14903 leader_election.cc:288] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203), 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461)
I20260926 02:19:42.301036 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5393cd7732ae49c2988f8a107df89c7c" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128727f8957d4944bba00c1e9b4357fd" is_pre_election: true
I20260926 02:19:42.301116 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5393cd7732ae49c2988f8a107df89c7c" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970821858a4d450980f4013a5e187c64" is_pre_election: true
I20260926 02:19:42.301267 14549 raft_consensus.cc:2454] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 0.
I20260926 02:19:42.301378 14694 raft_consensus.cc:2454] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 0.
I20260926 02:19:42.301805 14759 leader_election.cc:302] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 970821858a4d450980f4013a5e187c64, f1ad8edc866e4b66a759481586e32d5d; no voters:
I20260926 02:19:42.302106 14903 raft_consensus.cc:2781] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:42.302248 14903 raft_consensus.cc:482] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:42.302374 14903 raft_consensus.cc:3037] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:42.304548 14903 raft_consensus.cc:504] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.305140 14903 leader_election.cc:288] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 election: Requested vote from peers 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203), 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461)
I20260926 02:19:42.305619 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5393cd7732ae49c2988f8a107df89c7c" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128727f8957d4944bba00c1e9b4357fd"
I20260926 02:19:42.305608 14694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5393cd7732ae49c2988f8a107df89c7c" candidate_uuid: "f1ad8edc866e4b66a759481586e32d5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970821858a4d450980f4013a5e187c64"
I20260926 02:19:42.305836 14549 raft_consensus.cc:3037] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:42.305847 14694 raft_consensus.cc:3037] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:42.307927 14549 raft_consensus.cc:2454] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 1.
I20260926 02:19:42.308434 14761 leader_election.cc:302] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128727f8957d4944bba00c1e9b4357fd, f1ad8edc866e4b66a759481586e32d5d; no voters:
I20260926 02:19:42.308751 14694 raft_consensus.cc:2454] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1ad8edc866e4b66a759481586e32d5d in term 1.
I20260926 02:19:42.308741 14903 raft_consensus.cc:2781] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:42.309217 14903 raft_consensus.cc:686] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [term 1 LEADER]: Becoming Leader. State: Replica: f1ad8edc866e4b66a759481586e32d5d, State: Running, Role: LEADER
I20260926 02:19:42.309803 14903 consensus_queue.cc:237] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } }
I20260926 02:19:42.312107 14943 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: TxnStatusTablet state changed. Reason: New leader f1ad8edc866e4b66a759481586e32d5d. Latest consensus state: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.312517 14943 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:19:42.313326 14945 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:19:42.313560 14945 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:19:42.313795 14392 catalog_manager.cc:5495] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d reported cstate change: term changed from 0 to 1, leader changed from <none> to f1ad8edc866e4b66a759481586e32d5d (127.12.244.195). New cstate: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } health_report { overall_health: HEALTHY } } }
I20260926 02:19:42.367755 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.362505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.12.244.195:37693)
0926 02:19:42.362704 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.362712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.362738 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:19:42.363041 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.363051 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.363179 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.363271 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:42.363632 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.364435 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.365758 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.366279 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.366431 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.366895 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:42.367024 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:42.367178 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.367333 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.367416 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:19:42.367815 14937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.362382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:37693 (local address 127.0.0.1:50788)
0926 02:19:42.362667 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:42.362688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:42.362841 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:42.363312 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:42.363316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:42.363337 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:42.363495 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.363503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.364569 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.364577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.365620 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.365631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.366445 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.366451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.366627 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.366691 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:42.367177 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:42.367219 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:19:42.367310 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 02:19:42.367476 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:19:42.372912 14694 raft_consensus.cc:1261] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Refusing update from remote peer f1ad8edc866e4b66a759481586e32d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:42.372895 14549 raft_consensus.cc:1261] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Refusing update from remote peer f1ad8edc866e4b66a759481586e32d5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:42.374012 14943 consensus_queue.cc:1035] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 02:19:42.374292 14903 consensus_queue.cc:1035] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:42.379966 14905 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: TxnStatusTablet state changed. Reason: New leader f1ad8edc866e4b66a759481586e32d5d. Latest consensus state: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.380286 14905 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:42.380390 14904 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: TxnStatusTablet state changed. Reason: New leader f1ad8edc866e4b66a759481586e32d5d. Latest consensus state: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.380692 14904 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:42.384835 14943 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.385145 14943 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:19:42.386900 14949 mvcc.cc:205] Tried to move back new op lower bound from 7333434090992140288 to 7333434090745573376. Current Snapshot: MvccSnapshot[applied={T|T < 7333434090992140288}]
I20260926 02:19:42.387252 14903 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.387539 14903 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P f1ad8edc866e4b66a759481586e32d5d: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:19:42.398624 14904 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.399083 14904 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P 970821858a4d450980f4013a5e187c64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:42.399715 14905 tablet_replica.cc:439] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1ad8edc866e4b66a759481586e32d5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 } } peers { permanent_uuid: "970821858a4d450980f4013a5e187c64" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43461 } } peers { permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 } } }
I20260926 02:19:42.399971 14905 tablet_replica.cc:442] T 5393cd7732ae49c2988f8a107df89c7c P 128727f8957d4944bba00c1e9b4357fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:42.412595 14904 consensus_queue.cc:1035] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ad8edc866e4b66a759481586e32d5d" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:42.418184 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.409363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.12.244.194:43461)
0926 02:19:42.410866 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.410871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.410893 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:19:42.411010 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.411015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.411148 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.411334 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:42.411818 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.412966 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.415251 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.415948 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.416088 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.416649 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:42.417428 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:42.417579 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.417689 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.417781 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":626,"threads_started":1}
I20260926 02:19:42.421813 14930 consensus_queue.cc:1035] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "128727f8957d4944bba00c1e9b4357fd" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:42.438643 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.431262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:38055 (local address 127.12.244.195:37693)
0926 02:19:42.431437 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.431441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.432201 (+ 760us) server_negotiation.cc:407] Connection header received
0926 02:19:42.432454 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.432460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.432551 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.432657 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:42.434013 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.434757 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.436384 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.437279 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.437440 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.437907 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:42.438024 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:42.438124 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.438259 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.438340 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 02:19:42.438764 14964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.431125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:37693 (local address 127.12.244.194:38055)
0926 02:19:42.432082 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:42.432102 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:42.432275 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:42.433438 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:42.433442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:42.433478 (+ 36us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:42.433850 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.433861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.434913 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.434920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.436249 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.436261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.437475 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.437483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.437637 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.437700 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:42.438124 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:42.438137 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 02:19:42.438240 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:19:42.438383 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":637,"threads_started":1}
I20260926 02:19:42.487386 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.480152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:42305 (local address 127.12.244.194:43461)
0926 02:19:42.480355 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.480362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.481287 (+ 925us) server_negotiation.cc:407] Connection header received
0926 02:19:42.481580 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.481586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.481671 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.481732 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:42.482534 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.483388 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.485538 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.485982 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.486120 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.486596 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:42.486763 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:42.486833 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.486987 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.487080 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:19:42.487740 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.479280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:43461 (local address 127.12.244.195:42305)
0926 02:19:42.481185 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:42.481210 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:42.481413 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:42.481945 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:42.481950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:42.481963 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:42.482353 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.482364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.483543 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.483549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.485375 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:42.485390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.486144 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.486151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:42.486343 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.486427 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:42.486867 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:42.486879 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:19:42.486975 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 02:19:42.487178 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1701,"thread_start_us":1572,"threads_started":1}
I20260926 02:19:42.536736 14392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:42.537031 14392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:42.540863 14392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cab5f88b2c94a4cb1287edde68db8a3: Sending DeleteTablet for 3 replicas of tablet bed05bdc53534005b04507ea02bbfad1
I20260926 02:19:42.542856 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet bed05bdc53534005b04507ea02bbfad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:42 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:47542
I20260926 02:19:42.543340 14808 tablet_service.cc:1514] Processing DeleteTablet for tablet bed05bdc53534005b04507ea02bbfad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:42 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:50770
I20260926 02:19:42.543785 14669 tablet_service.cc:1514] Processing DeleteTablet for tablet bed05bdc53534005b04507ea02bbfad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:42 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:39040
I20260926 02:19:42.564167 14973 tablet_replica.cc:330] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: stopping tablet replica
I20260926 02:19:42.564646 14973 raft_consensus.cc:2229] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:19:42.565292 14973 raft_consensus.cc:2258] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:42.565937 14971 tablet_replica.cc:330] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: stopping tablet replica
I20260926 02:19:42.566154 14971 raft_consensus.cc:2229] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:42.566438 14971 raft_consensus.cc:2258] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:42.567080 14972 tablet_replica.cc:330] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: stopping tablet replica
I20260926 02:19:42.567373 14972 raft_consensus.cc:2229] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:42.567726 14972 raft_consensus.cc:2258] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:42.569000 14973 ts_tablet_manager.cc:1905] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:42.576257 14972 ts_tablet_manager.cc:1905] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:42.583971 14971 ts_tablet_manager.cc:1905] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:19:42 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:19:42.594074 14972 ts_tablet_manager.cc:1918] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:42.594245 14972 log.cc:1196] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/bed05bdc53534005b04507ea02bbfad1
I20260926 02:19:42.594758 14972 ts_tablet_manager.cc:1939] T bed05bdc53534005b04507ea02bbfad1 P f1ad8edc866e4b66a759481586e32d5d: Deleting consensus metadata
I20260926 02:19:42.596196 14378 catalog_manager.cc:4841] TS f1ad8edc866e4b66a759481586e32d5d (127.12.244.195:37693): tablet bed05bdc53534005b04507ea02bbfad1 (table test-table [id=17aa8beb0d144cb386797efbd90e9a96]) successfully deleted
I20260926 02:19:42.597168 14971 ts_tablet_manager.cc:1918] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:42.597352 14971 log.cc:1196] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/bed05bdc53534005b04507ea02bbfad1
I20260926 02:19:42.597872 14971 ts_tablet_manager.cc:1939] T bed05bdc53534005b04507ea02bbfad1 P 128727f8957d4944bba00c1e9b4357fd: Deleting consensus metadata
I20260926 02:19:42.599047 14378 catalog_manager.cc:4841] TS 128727f8957d4944bba00c1e9b4357fd (127.12.244.193:45203): tablet bed05bdc53534005b04507ea02bbfad1 (table test-table [id=17aa8beb0d144cb386797efbd90e9a96]) successfully deleted
I20260926 02:19:42.599119 14973 ts_tablet_manager.cc:1918] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:42.599267 14973 log.cc:1196] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/bed05bdc53534005b04507ea02bbfad1
I20260926 02:19:42.599737 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.566327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.12.244.193:45203)
0926 02:19:42.575412 (+ 9085us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.575419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.575444 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:19:42.575508 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.575514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.575593 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.575698 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:42.577814 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.579050 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.581113 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.581773 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.581906 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.588176 (+ 6270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:42.588193 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:42.588214 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:42.588266 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:42.592471 (+ 4205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.594033 (+ 1562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.594044 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.594048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.594231 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.594765 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.594770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.594774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.595330 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:42.595631 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.596860 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.597107 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8956,"thread_start_us":8764,"threads_started":1}
I20260926 02:19:42.599967 14973 ts_tablet_manager.cc:1939] T bed05bdc53534005b04507ea02bbfad1 P 970821858a4d450980f4013a5e187c64: Deleting consensus metadata
I20260926 02:19:42.601023 14376 catalog_manager.cc:4841] TS 970821858a4d450980f4013a5e187c64 (127.12.244.194:43461): tablet bed05bdc53534005b04507ea02bbfad1 (table test-table [id=17aa8beb0d144cb386797efbd90e9a96]) successfully deleted
W20260926 02:19:42.601011 14594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47548
I20260926 02:19:42.623387 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.604952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.12.244.254:33475)
0926 02:19:42.605195 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.605202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.605728 (+ 526us) server_negotiation.cc:407] Connection header received
0926 02:19:42.605916 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.605921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.606009 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.606122 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:42.607918 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.609067 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.610934 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.611735 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.611904 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.615697 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:42.615706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:42.615711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:42.615765 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:42.620003 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.621048 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.621056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.621060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.621226 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.621689 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.621694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.621698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.622076 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:42.622369 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.622734 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.622988 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260926 02:19:42.624720 14392 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57094
I20260926 02:19:42.627341 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14463
I20260926 02:19:42.647046 14975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.629522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47554 (local address 127.12.244.193:45203)
0926 02:19:42.629880 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.629888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.630098 (+ 210us) server_negotiation.cc:407] Connection header received
0926 02:19:42.630389 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.630396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.630489 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.630599 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:42.632268 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.633404 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.635811 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.636523 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.636682 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.639755 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:42.639763 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:42.639768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:42.639815 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:42.643727 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.644677 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.644683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.644688 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.644861 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.645382 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.645387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.645391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.645732 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:42.645984 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.646417 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.646683 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
W20260926 02:19:42.647895 14594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47554
W20260926 02:19:42.648634 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14463: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:42.648846 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 14463
I20260926 02:19:42.674104 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14603
Sep 26 02:19:42 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:19:42.698490 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.675494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.12.244.194:43461)
0926 02:19:42.675682 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.675688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.675707 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:19:42.675776 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.675781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.675856 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.675969 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:42.677799 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.679263 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.681332 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.682013 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.682176 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.688646 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:42.688668 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:42.688701 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:42.688771 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:42.693533 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.694665 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.694677 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.694681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.694890 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.695443 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.695449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.695453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.695967 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:42.696345 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.697805 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.698078 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 02:19:42.699606 14734 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39054
W20260926 02:19:42.700369 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14603: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:42.700552 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 14603
I20260926 02:19:42.727562 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14743
Sep 26 02:19:42 dist-test-slave-qfmw krb5kdc[14347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:19:42.746071 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:42.728491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.12.244.195:37693)
0926 02:19:42.728755 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:42.728762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:42.728783 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:19:42.728949 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:42.728954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:42.729045 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:42.729165 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:42.730456 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.731701 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.733278 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:42.733748 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:42.733899 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:42.739066 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:42.739083 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:42.739155 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:42.739211 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:42.742583 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.743570 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.743580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.743582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.743750 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:42.744185 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:42.744204 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:42.744206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:42.744677 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:42.745187 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:42.745404 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 02:19:42.745672 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
W20260926 02:19:42.747351 14871 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50802
W20260926 02:19:42.748152 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14743: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:42.748358 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 14743
I20260926 02:19:42.776728 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14360
I20260926 02:19:42.778093 14432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:42.907755 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 14360
2026-09-26T02:19:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5161 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:43 dist-test-slave-qfmw krb5kdc[14983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:43 dist-test-slave-qfmw krb5kdc[14983](info): set up 2 sockets
Sep 26 02:19:43 dist-test-slave-qfmw krb5kdc[14983](info): commencing operation
krb5kdc: starting...
W20260926 02:19:44.055136 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:19:44 dist-test-slave-qfmw krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:19:44.262791 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45759
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:45759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:44.609834 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:44.610265 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:44.610452 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:44.618012 14996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:44.618203 14996 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:19:44.618287 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:44.618408 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:44.618505 14996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:44.618597 14996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:44.628952 14996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:45759
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:44.630409 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:44.631876 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:44.644966 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:44.645018 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:44.644905 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:44.645794 14996 server_base.cc:1034] running on GCE node
I20260926 02:19:44.646788 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:44.648538 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:44.649842 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790389184649775 us; error 43 us; skew 500 ppm
I20260926 02:19:44.650260 14996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:44 dist-test-slave-qfmw krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:44.653878 14996 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:19:44.655747 14996 webserver.cc:460] Webserver started at http://127.12.244.254:46871/ using document root <none> and password file <none>
I20260926 02:19:44.656540 14996 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:44.656678 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:44.657003 14996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:44.660240 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "c275afc36a7542f5b2480ede5e930b3a"
format_stamp: "Formatted at 2026-09-26 02:19:44 on dist-test-slave-qfmw"
server_key: "289045d130c5893eaf4c89272e61aef6"
server_key_iv: "6f939fd130e6ce894eb8604438f4c285"
server_key_version: "encryptionkey@0"
I20260926 02:19:44.661134 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "c275afc36a7542f5b2480ede5e930b3a"
format_stamp: "Formatted at 2026-09-26 02:19:44 on dist-test-slave-qfmw"
server_key: "289045d130c5893eaf4c89272e61aef6"
server_key_iv: "6f939fd130e6ce894eb8604438f4c285"
server_key_version: "encryptionkey@0"
I20260926 02:19:44.667797 14996 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:19:44.671998 15011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:44.673681 14996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:19:44.673947 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "c275afc36a7542f5b2480ede5e930b3a"
format_stamp: "Formatted at 2026-09-26 02:19:44 on dist-test-slave-qfmw"
server_key: "289045d130c5893eaf4c89272e61aef6"
server_key_iv: "6f939fd130e6ce894eb8604438f4c285"
server_key_version: "encryptionkey@0"
I20260926 02:19:44.674201 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:44.691860 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:44.698194 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:44.698544 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:44.724660 14996 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:45759
I20260926 02:19:44.724668 15072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:45759 every 8 connection(s)
I20260926 02:19:44.726190 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:44.730439 15073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:44.733456 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 14996
I20260926 02:19:44.733657 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:19:44.734455 13267 external_mini_cluster.cc:1428] Setting key 02ba6ffb1aefa3148566a30d044b84dc
I20260926 02:19:44.743151 15073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Bootstrap starting.
I20260926 02:19:44.746577 15073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:44.747745 15073 log.cc:824] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:19:44 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:44.751008 15073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: No bootstrap required, opened a new log
I20260926 02:19:44.758448 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:44.736949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.12.244.254:45759)
0926 02:19:44.737802 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:44.737816 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:44.737863 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:19:44.739686 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:44.739712 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:44.740162 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:44.740622 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:44.742099 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:44.743597 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:44.744892 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:44.745616 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:44.745828 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:44.750424 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:44.750433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:44.750449 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:44.750487 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:44.753734 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:44.754505 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:44.754562 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:44.754573 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:44.754779 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:44.755231 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:44.755237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:44.755240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:44.755932 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:44.756275 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:44.756752 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 02:19:44.757224 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":384,"threads_started":1}
I20260926 02:19:44.759778 15073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c275afc36a7542f5b2480ede5e930b3a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45759 } }
I20260926 02:19:44.760227 15073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:44.760363 15073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c275afc36a7542f5b2480ede5e930b3a, State: Initialized, Role: FOLLOWER
I20260926 02:19:44.760987 15073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c275afc36a7542f5b2480ede5e930b3a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45759 } }
I20260926 02:19:44.761242 15073 raft_consensus.cc:388] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:44.761395 15073 raft_consensus.cc:482] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:44.761587 15073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:44.763933 15073 raft_consensus.cc:504] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c275afc36a7542f5b2480ede5e930b3a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45759 } }
I20260926 02:19:44.764482 15073 leader_election.cc:302] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c275afc36a7542f5b2480ede5e930b3a; no voters:
I20260926 02:19:44.765067 15073 leader_election.cc:288] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:44.765209 15078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:44.766636 15078 raft_consensus.cc:686] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [term 1 LEADER]: Becoming Leader. State: Replica: c275afc36a7542f5b2480ede5e930b3a, State: Running, Role: LEADER
I20260926 02:19:44.767246 15078 consensus_queue.cc:237] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c275afc36a7542f5b2480ede5e930b3a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45759 } }
I20260926 02:19:44.767553 15073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:44.770454 15080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c275afc36a7542f5b2480ede5e930b3a. Latest consensus state: current_term: 1 leader_uuid: "c275afc36a7542f5b2480ede5e930b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c275afc36a7542f5b2480ede5e930b3a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45759 } } }
I20260926 02:19:44.770979 15080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:44.771273 15079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c275afc36a7542f5b2480ede5e930b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c275afc36a7542f5b2480ede5e930b3a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45759 } } }
I20260926 02:19:44.771559 15079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:44.774137 15087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:44.778697 15087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:44.787832 15087 catalog_manager.cc:1347] Generated new cluster ID: 3bcff9d5e0f34a8baabae139a3b851e5
I20260926 02:19:44.787959 15087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:44.807679 15087 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:44.809641 15087 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:44.821274 15087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Generated new TSK 0
I20260926 02:19:44.822475 15087 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:19:44.875726 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45759
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:45.227555 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:45.227998 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:45.228176 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:45.235816 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:45.235996 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:45.236153 15099 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:19:45.236240 15099 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:19:45.236373 15099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:19:45.246953 15099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:45759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:45.248389 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:45.249982 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:45.265576 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:45.265630 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:45.265576 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:45.266826 15099 server_base.cc:1034] running on GCE node
I20260926 02:19:45.267628 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:45.268807 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:45.270102 15099 hybrid_clock.cc:648] HybridClock initialized: now 1790389185270006 us; error 97 us; skew 500 ppm
I20260926 02:19:45.270567 15099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:45 dist-test-slave-qfmw krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:45.278865 15099 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:19:45.280853 15099 webserver.cc:460] Webserver started at http://127.12.244.193:44815/ using document root <none> and password file <none>
I20260926 02:19:45.281694 15099 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:45.281863 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:45.282222 15099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:45.285691 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "9607ad0bc7a146fcb0ca111e5b57f23b"
format_stamp: "Formatted at 2026-09-26 02:19:45 on dist-test-slave-qfmw"
server_key: "7827688f5f26fb31f59eca23e2de3a99"
server_key_iv: "26ed6cb0ca49d566a758f57db44333c6"
server_key_version: "encryptionkey@0"
I20260926 02:19:45.286641 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "9607ad0bc7a146fcb0ca111e5b57f23b"
format_stamp: "Formatted at 2026-09-26 02:19:45 on dist-test-slave-qfmw"
server_key: "7827688f5f26fb31f59eca23e2de3a99"
server_key_iv: "26ed6cb0ca49d566a758f57db44333c6"
server_key_version: "encryptionkey@0"
I20260926 02:19:45.292721 15099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:19:45.297107 15114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:45.298835 15099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 02:19:45.299137 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "9607ad0bc7a146fcb0ca111e5b57f23b"
format_stamp: "Formatted at 2026-09-26 02:19:45 on dist-test-slave-qfmw"
server_key: "7827688f5f26fb31f59eca23e2de3a99"
server_key_iv: "26ed6cb0ca49d566a758f57db44333c6"
server_key_version: "encryptionkey@0"
I20260926 02:19:45.299398 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:45.323657 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:45.328486 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:45.328801 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:45.331661 15099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:45.331825 15099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:45.331992 15099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:45.332087 15099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:19:45 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:45.361995 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.332633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:55193 (local address 127.12.244.254:45759)
0926 02:19:45.333068 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:45.333075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:45.334601 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 02:19:45.337657 (+ 3056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:45.337666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:45.337784 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:45.337916 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:45.340699 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.342038 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.344900 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.345507 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.345685 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.352611 (+ 6926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:45.352622 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:45.352629 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:45.352687 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:45.357097 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.358471 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.358478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.358482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.358642 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.359337 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.359343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.359346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.359762 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:45.359921 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:45.361096 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0926 02:19:45.361445 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":215,"threads_started":1}
I20260926 02:19:45.362663 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.333425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45759 (local address 127.12.244.193:55193)
0926 02:19:45.334368 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:45.334438 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:45.337224 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:45.338297 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:45.338309 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:45.339294 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:45.340410 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.340439 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.342227 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.342232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.344681 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.344706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.345890 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.345906 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.346357 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.347484 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:45.347542 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:45.352069 (+ 4527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:45.357419 (+ 5350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.357434 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.357467 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.358287 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.358784 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.358791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.358798 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.359188 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.359892 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:45.359901 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:45.360519 (+ 618us) client_negotiation.cc:769] Sending connection context
0926 02:19:45.360960 (+ 441us) client_negotiation.cc:240] Negotiation successful
0926 02:19:45.361463 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":405,"threads_started":1}
I20260926 02:19:45.379794 15099 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:44675
I20260926 02:19:45.379858 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:44675 every 8 connection(s)
I20260926 02:19:45.381250 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:19:45.383656 15235 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45759
I20260926 02:19:45.383915 15235 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:45.384795 15235 heartbeater.cc:511] Master 127.12.244.254:45759 requested a full tablet report, sending...
I20260926 02:19:45.387547 15028 ts_manager.cc:194] Registered new tserver with Master: 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675)
I20260926 02:19:45.388267 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 15099
I20260926 02:19:45.388513 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:19:45.389331 13267 external_mini_cluster.cc:1428] Setting key 520d42a5750cd11bdfb4e009c8f410b3
I20260926 02:19:45.389400 15028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:55193
I20260926 02:19:45.390532 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.376061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45759 (local address 127.12.244.193:39691)
0926 02:19:45.376723 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:45.376753 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:45.376939 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:45.377352 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:45.377357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:45.377955 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:45.378413 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.378426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.379569 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.379575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.380763 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.380776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.381595 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.381602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.381845 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.382669 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:45.382699 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:45.383833 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:45.387312 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.387317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.387321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.387898 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.388407 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.388420 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.388423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.388594 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.389301 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:45.389309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:45.389522 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 02:19:45.389702 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 02:19:45.390046 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":357,"threads_started":1}
I20260926 02:19:45.391276 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.376022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:39691 (local address 127.12.244.254:45759)
0926 02:19:45.376215 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:45.376220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:45.376854 (+ 634us) server_negotiation.cc:407] Connection header received
0926 02:19:45.377097 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:45.377102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:45.377214 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:45.377325 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:45.378563 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.379425 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.380917 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.381348 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.381488 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.384055 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:45.384064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:45.384070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:45.384111 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:45.387170 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.388038 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.388046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.388050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.388258 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.388742 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.388748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.388756 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.389164 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:45.389329 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:45.389818 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 02:19:45.390072 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:19:45.435073 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45759
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:45.783133 15239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:45.783449 15239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:45.783665 15239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:45.791014 15239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:45.791162 15239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:45.791332 15239 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:19:45.791640 15239 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:19:45.791817 15239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:19:45.802737 15239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:45759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:45.804324 15239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:45.805820 15239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:45.820814 15244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:45.820760 15245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:45.820730 15247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:45.821717 15239 server_base.cc:1034] running on GCE node
I20260926 02:19:45.822381 15239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:45.823448 15239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:45.824687 15239 hybrid_clock.cc:648] HybridClock initialized: now 1790389185824635 us; error 36 us; skew 500 ppm
I20260926 02:19:45.825048 15239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:45 dist-test-slave-qfmw krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:45.828511 15239 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:19:45.830238 15239 webserver.cc:460] Webserver started at http://127.12.244.194:42675/ using document root <none> and password file <none>
I20260926 02:19:45.831028 15239 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:45.831236 15239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:45.831584 15239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:45.834893 15239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "521378b341fb499cb4a56e5567691f79"
format_stamp: "Formatted at 2026-09-26 02:19:45 on dist-test-slave-qfmw"
server_key: "672d3577c3f333ea852502e486a6230f"
server_key_iv: "cd49d11470f950fc623dc17604bd0e70"
server_key_version: "encryptionkey@0"
I20260926 02:19:45.835796 15239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "521378b341fb499cb4a56e5567691f79"
format_stamp: "Formatted at 2026-09-26 02:19:45 on dist-test-slave-qfmw"
server_key: "672d3577c3f333ea852502e486a6230f"
server_key_iv: "cd49d11470f950fc623dc17604bd0e70"
server_key_version: "encryptionkey@0"
I20260926 02:19:45.841468 15239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:19:45.845451 15254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:45.847029 15239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:19:45.847271 15239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "521378b341fb499cb4a56e5567691f79"
format_stamp: "Formatted at 2026-09-26 02:19:45 on dist-test-slave-qfmw"
server_key: "672d3577c3f333ea852502e486a6230f"
server_key_iv: "cd49d11470f950fc623dc17604bd0e70"
server_key_version: "encryptionkey@0"
I20260926 02:19:45.847514 15239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:45.867297 15239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:45.872392 15239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:45.872714 15239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:45.875560 15239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:45.875702 15239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:45.875860 15239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:45.875963 15239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:19:45 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:45.904029 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.876465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:54821 (local address 127.12.244.254:45759)
0926 02:19:45.876653 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:45.876659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:45.878234 (+ 1575us) server_negotiation.cc:407] Connection header received
0926 02:19:45.880932 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:45.880938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:45.881033 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:45.881155 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:45.883814 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.884869 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.886771 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.887498 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.887657 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.894367 (+ 6710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:45.894377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:45.894384 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:45.894433 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:45.898205 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.899131 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.899141 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.899144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.899320 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.900106 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.900111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.900113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.900544 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:45.900718 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:45.903346 (+ 2628us) server_negotiation.cc:299] Negotiation successful
0926 02:19:45.903607 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:19:45.904433 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.877089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45759 (local address 127.12.244.194:54821)
0926 02:19:45.878034 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:45.878087 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:45.880562 (+ 2475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:45.881480 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:45.881497 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:45.882396 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:45.883445 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.883481 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.885056 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.885071 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.886610 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.886623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.887824 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.887835 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.888138 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.889150 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:45.889197 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:45.894031 (+ 4834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:45.898393 (+ 4362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.898403 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.898427 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.898928 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.899478 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.899487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.899494 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.899899 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.900814 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:45.900823 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:45.901495 (+ 672us) client_negotiation.cc:769] Sending connection context
0926 02:19:45.901935 (+ 440us) client_negotiation.cc:240] Negotiation successful
0926 02:19:45.902594 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":406,"threads_started":1}
I20260926 02:19:45.921844 15239 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:45861
I20260926 02:19:45.921885 15373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:45861 every 8 connection(s)
I20260926 02:19:45.923512 15239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:19:45.925940 15374 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45759
I20260926 02:19:45.926199 15374 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:45.926975 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 15239
I20260926 02:19:45.927181 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:19:45.927161 15374 heartbeater.cc:511] Master 127.12.244.254:45759 requested a full tablet report, sending...
I20260926 02:19:45.927735 13267 external_mini_cluster.cc:1428] Setting key 4d071f5de9d919c0af0f28ceac8c0925
I20260926 02:19:45.928733 15027 ts_manager.cc:194] Registered new tserver with Master: 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861)
I20260926 02:19:45.929037 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.912801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:38073 (local address 127.12.244.254:45759)
0926 02:19:45.913026 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:45.913036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:45.913406 (+ 370us) server_negotiation.cc:407] Connection header received
0926 02:19:45.913649 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:45.913659 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:45.913724 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:45.913831 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:45.915129 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.916140 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.918919 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.919550 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.919715 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.922095 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:45.922105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:45.922110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:45.922155 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:45.925646 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.926566 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.926573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.926578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.926762 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:45.927240 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:45.927249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:45.927252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:45.927743 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:45.927898 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:45.928372 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 02:19:45.928613 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:19:45.929081 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:45.912279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45759 (local address 127.12.244.194:38073)
0926 02:19:45.913260 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:45.913285 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:45.913507 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:45.913862 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:45.913867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:45.914355 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:45.914851 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.914863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.916351 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.916361 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.918660 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:45.918683 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:45.919775 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:45.919780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:45.919988 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:45.920919 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:45.920944 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:45.921772 (+ 828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:45.925803 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.925808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.925815 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.926426 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.926928 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:45.926933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:45.926936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:45.927123 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:45.927900 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:45.927905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:45.928117 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 02:19:45.928275 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 02:19:45.928611 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":588,"threads_started":1}
I20260926 02:19:45.929992 15027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:54821
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:19:45.972816 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45759
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:46.328078 15378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:46.328382 15378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:46.328487 15378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:46.335983 15378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:46.336117 15378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:46.336205 15378 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:19:46.336259 15378 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:19:46.336330 15378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:19:46.346344 15378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:45759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:46.347668 15378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:46.349025 15378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:46.363193 15383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:46.363155 15386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:46.363124 15384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:46.364140 15378 server_base.cc:1034] running on GCE node
I20260926 02:19:46.364812 15378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:46.365911 15378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:46.367213 15378 hybrid_clock.cc:648] HybridClock initialized: now 1790389186367142 us; error 55 us; skew 500 ppm
I20260926 02:19:46.367585 15378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:46 dist-test-slave-qfmw krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389186, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:46.371405 15378 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:19:46.373243 15378 webserver.cc:460] Webserver started at http://127.12.244.195:34717/ using document root <none> and password file <none>
I20260926 02:19:46.374014 15378 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:46.374168 15378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:46.374516 15378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:46.378072 15378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "fbd90f0fc17e48ad951a9e043ddcf879"
format_stamp: "Formatted at 2026-09-26 02:19:46 on dist-test-slave-qfmw"
server_key: "6f19184f6faaf48da115fddf8cc8996d"
server_key_iv: "3001334fb6c8592dea6c4182ebd2f153"
server_key_version: "encryptionkey@0"
I20260926 02:19:46.378993 15378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "fbd90f0fc17e48ad951a9e043ddcf879"
format_stamp: "Formatted at 2026-09-26 02:19:46 on dist-test-slave-qfmw"
server_key: "6f19184f6faaf48da115fddf8cc8996d"
server_key_iv: "3001334fb6c8592dea6c4182ebd2f153"
server_key_version: "encryptionkey@0"
I20260926 02:19:46.385257 15378 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 02:19:46.389474 15393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:46.391249 15378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:19:46.391456 15378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "fbd90f0fc17e48ad951a9e043ddcf879"
format_stamp: "Formatted at 2026-09-26 02:19:46 on dist-test-slave-qfmw"
server_key: "6f19184f6faaf48da115fddf8cc8996d"
server_key_iv: "3001334fb6c8592dea6c4182ebd2f153"
server_key_version: "encryptionkey@0"
I20260926 02:19:46.391695 15378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:46.393584 15235 heartbeater.cc:503] Master 127.12.244.254:45759 was elected leader, sending a full tablet report...
I20260926 02:19:46.414443 15378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:46.419334 15378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:46.419641 15378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:46.422446 15378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:46.422590 15378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:46.422729 15378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:46.422811 15378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:19:46 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389186, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:46.448832 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.423356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:55393 (local address 127.12.244.254:45759)
0926 02:19:46.423557 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.423562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.425518 (+ 1956us) server_negotiation.cc:407] Connection header received
0926 02:19:46.427985 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.427991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.428084 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.428213 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:46.430862 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.432231 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.434603 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.435180 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.435339 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.441376 (+ 6037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:46.441383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:46.441388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:46.441447 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:46.445114 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.445973 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.445979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.445983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.446142 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.446618 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.446625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.446628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.446920 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:46.447111 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.448196 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.448455 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:19:46.449983 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.424325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45759 (local address 127.12.244.195:55393)
0926 02:19:46.425289 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.425393 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.427564 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.428536 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.428552 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.429514 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:46.430537 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.430561 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.432450 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.432461 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.434422 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.434435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.435495 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.435505 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.435907 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.436978 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:46.437027 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:46.440979 (+ 3952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:46.445297 (+ 4318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:46.445309 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:46.445337 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:46.445812 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:46.446275 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:46.446280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:46.446284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:46.446501 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:46.447061 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:46.447070 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:46.447636 (+ 566us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.448075 (+ 439us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.448660 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":417,"threads_started":1}
I20260926 02:19:46.474576 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.457602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:52523 (local address 127.12.244.254:45759)
0926 02:19:46.457836 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.457844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.458420 (+ 576us) server_negotiation.cc:407] Connection header received
0926 02:19:46.458734 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.458740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.458841 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.458945 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:46.460399 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.461393 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.464220 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.464877 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.465041 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.468040 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:46.468045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:46.468048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:46.468082 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:46.471018 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.472152 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.472161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.472165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.472322 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.472899 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.472917 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.472920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.473273 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:46.473447 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.474121 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.474322 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:19:46.475126 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.457454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45759 (local address 127.12.244.195:52523)
0926 02:19:46.458278 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.458303 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.458507 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.459228 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.459237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.459946 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:46.460236 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.460246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.461577 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.461584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.464032 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.464052 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.465142 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.465152 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.465442 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.466303 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:46.466335 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:46.467782 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:46.471192 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:46.471203 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:46.471206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:46.471972 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:46.472464 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:46.472470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:46.472473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:46.472725 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:46.473426 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:46.473439 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:46.473716 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.473954 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.474420 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":374,"threads_started":1}
I20260926 02:19:46.486042 15378 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:39559
I20260926 02:19:46.486078 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:39559 every 8 connection(s)
I20260926 02:19:46.487504 15378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:19:46.490072 15513 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45759
I20260926 02:19:46.490286 15513 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:46.490911 15513 heartbeater.cc:511] Master 127.12.244.254:45759 requested a full tablet report, sending...
I20260926 02:19:46.492497 15028 ts_manager.cc:194] Registered new tserver with Master: fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559)
I20260926 02:19:46.493331 15028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:55393
I20260926 02:19:46.494920 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 15378
I20260926 02:19:46.495172 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:19:46.495635 13267 external_mini_cluster.cc:1428] Setting key 453332654580dea78b3fd7f5a6e2b347
I20260926 02:19:46.498935 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:19:46 dist-test-slave-qfmw krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:19:46 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:46.547528 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.532906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.12.244.254:45759)
0926 02:19:46.533167 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.533174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.533392 (+ 218us) server_negotiation.cc:407] Connection header received
0926 02:19:46.533568 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.533572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.533635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.533723 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:46.535312 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.536083 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.537368 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.537786 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.537889 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.542021 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:46.542026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:46.542030 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:46.542068 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:46.544778 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.545461 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.545467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.545471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.545614 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.546002 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.546015 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.546020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.546293 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:46.546553 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.546880 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.547164 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 02:19:46.552045 15028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:19:46.558812 15028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:46.595767 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.574092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.12.244.194:45861)
0926 02:19:46.585879 (+ 11787us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.585893 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.586898 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 02:19:46.587078 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.587119 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.587500 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.587758 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:46.588500 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.589889 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.592375 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.593893 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.594060 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.594255 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.595218 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.595425 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11567,"thread_start_us":11393,"threads_started":1}
I20260926 02:19:46.595852 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.574041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.12.244.195:39559)
0926 02:19:46.585527 (+ 11486us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.585540 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.587889 (+ 2349us) server_negotiation.cc:407] Connection header received
0926 02:19:46.588015 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.588022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.588281 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.588478 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:46.589385 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.591146 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.593700 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.594804 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.595030 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.595241 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.595379 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.595532 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11159,"thread_start_us":10930,"threads_started":1}
I20260926 02:19:46.595929 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.573661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:39559 (local address 127.0.0.1:57942)
0926 02:19:46.587588 (+ 13927us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.587610 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.587752 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.588538 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.588542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.588606 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:46.589169 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.589182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.591290 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.591296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.593499 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.593510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.595025 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.595031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.595224 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.595256 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.595343 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.595486 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13798,"mutex_wait_us":80,"spinlock_wait_cycles":54656,"thread_start_us":12473,"threads_started":1}
I20260926 02:19:46.596436 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.572075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.0.0.1:48790)
0926 02:19:46.573049 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.573114 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.573348 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.588498 (+ 15150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.588514 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.588546 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:46.589129 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.589148 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.591353 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.591360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.593600 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.593614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.595524 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.595530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.595714 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.595747 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.595820 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.595988 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":136,"thread_start_us":393,"threads_started":1}
I20260926 02:19:46.596872 15529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.572768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.0.0.1:50124)
0926 02:19:46.586505 (+ 13737us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.586534 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.586727 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.588002 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.588008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.588033 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:46.588323 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.588332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.590054 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.590061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.592191 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.592205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.594380 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.594386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.594601 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.594657 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.594805 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.595010 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13556,"mutex_wait_us":1,"thread_start_us":14545,"threads_started":1}
I20260926 02:19:46.597570 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.572234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.12.244.193:44675)
0926 02:19:46.587914 (+ 15680us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.587924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.587945 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:19:46.588023 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.588029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.588268 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.589379 (+ 1111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:46.589611 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.591192 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.593797 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.595296 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.596846 (+ 1550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.597054 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.597131 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.597307 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15524,"thread_start_us":15402,"threads_started":1}
I20260926 02:19:46.600427 15443 tablet_service.cc:1467] Processing CreateTablet for tablet ebab614c85484bb3bb37cb0b08731871 (DEFAULT_TABLE table=test-table [id=9d60adf985494cd681878ba3287407b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:46.600437 15304 tablet_service.cc:1467] Processing CreateTablet for tablet ebab614c85484bb3bb37cb0b08731871 (DEFAULT_TABLE table=test-table [id=9d60adf985494cd681878ba3287407b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:46.601140 15165 tablet_service.cc:1467] Processing CreateTablet for tablet ebab614c85484bb3bb37cb0b08731871 (DEFAULT_TABLE table=test-table [id=9d60adf985494cd681878ba3287407b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:46.601949 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebab614c85484bb3bb37cb0b08731871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:46.602049 15443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebab614c85484bb3bb37cb0b08731871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:46.602375 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebab614c85484bb3bb37cb0b08731871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:46.616150 15532 tablet_bootstrap.cc:492] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Bootstrap starting.
I20260926 02:19:46.620322 15532 tablet_bootstrap.cc:654] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:46.621742 15533 tablet_bootstrap.cc:492] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Bootstrap starting.
I20260926 02:19:46.621943 15534 tablet_bootstrap.cc:492] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Bootstrap starting.
I20260926 02:19:46.621969 15532 log.cc:824] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:46.625097 15532 tablet_bootstrap.cc:492] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: No bootstrap required, opened a new log
I20260926 02:19:46.625433 15532 ts_tablet_manager.cc:1397] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 02:19:46.626281 15533 tablet_bootstrap.cc:654] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:46.626559 15534 tablet_bootstrap.cc:654] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:46.627759 15533 log.cc:824] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:46.627759 15534 log.cc:824] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:46.631464 15533 tablet_bootstrap.cc:492] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: No bootstrap required, opened a new log
I20260926 02:19:46.631738 15534 tablet_bootstrap.cc:492] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: No bootstrap required, opened a new log
I20260926 02:19:46.631778 15533 ts_tablet_manager.cc:1397] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 02:19:46.632025 15534 ts_tablet_manager.cc:1397] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 02:19:46.635825 15532 raft_consensus.cc:348] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.636368 15532 raft_consensus.cc:374] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:46.636540 15532 raft_consensus.cc:729] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9607ad0bc7a146fcb0ca111e5b57f23b, State: Initialized, Role: FOLLOWER
I20260926 02:19:46.637238 15532 consensus_queue.cc:260] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.639230 15532 ts_tablet_manager.cc:1428] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260926 02:19:46.640863 15533 raft_consensus.cc:348] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.641448 15533 raft_consensus.cc:374] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:46.641598 15533 raft_consensus.cc:729] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521378b341fb499cb4a56e5567691f79, State: Initialized, Role: FOLLOWER
I20260926 02:19:46.642280 15533 consensus_queue.cc:260] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.642480 15534 raft_consensus.cc:348] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.642990 15534 raft_consensus.cc:374] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:46.643105 15534 raft_consensus.cc:729] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd90f0fc17e48ad951a9e043ddcf879, State: Initialized, Role: FOLLOWER
I20260926 02:19:46.643709 15534 consensus_queue.cc:260] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.645080 15513 heartbeater.cc:503] Master 127.12.244.254:45759 was elected leader, sending a full tablet report...
I20260926 02:19:46.645776 15534 ts_tablet_manager.cc:1428] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 02:19:46.646948 15374 heartbeater.cc:503] Master 127.12.244.254:45759 was elected leader, sending a full tablet report...
I20260926 02:19:46.647745 15533 ts_tablet_manager.cc:1428] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Time spent starting tablet: real 0.016s user 0.003s sys 0.011s
W20260926 02:19:46.675825 15375 tablet.cc:2369] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:19:46.739262 15514 tablet.cc:2369] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:46.860932 15539 raft_consensus.cc:482] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:46.861279 15539 raft_consensus.cc:504] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.863184 15539 leader_election.cc:288] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675), fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559)
I20260926 02:19:46.870568 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.863702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:48965 (local address 127.12.244.193:44675)
0926 02:19:46.864014 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.864021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.864540 (+ 519us) server_negotiation.cc:407] Connection header received
0926 02:19:46.864798 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.864804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.864889 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.865017 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:46.865723 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.866506 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.868782 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.869543 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.869745 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.869823 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.870129 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.870280 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:19:46.870682 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.863580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.12.244.194:48965)
0926 02:19:46.864353 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.864433 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.864640 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.865075 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.865081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.865118 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:46.865570 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.865582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.866650 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.866659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.868618 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.868631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.869760 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.869766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.869952 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.869976 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.870102 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.870243 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":31,"thread_start_us":402,"threads_started":1}
I20260926 02:19:46.870870 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.864712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:39419 (local address 127.12.244.195:39559)
0926 02:19:46.864916 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.864925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.865384 (+ 459us) server_negotiation.cc:407] Connection header received
0926 02:19:46.865614 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.865618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.865673 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.865763 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:46.866291 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.866989 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.869088 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.869904 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.870080 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.870157 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.870477 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.870608 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:19:46.871129 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.864453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:39559 (local address 127.12.244.194:39419)
0926 02:19:46.865272 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.865287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.865447 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.865786 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.865790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.865827 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:46.866143 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.866151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.867144 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.867152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.868865 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.868943 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.870112 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.870118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.870299 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.870332 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.870446 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.870577 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":35,"thread_start_us":505,"threads_started":1}
I20260926 02:19:46.871992 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebab614c85484bb3bb37cb0b08731871" candidate_uuid: "521378b341fb499cb4a56e5567691f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" is_pre_election: true
I20260926 02:19:46.872326 15468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebab614c85484bb3bb37cb0b08731871" candidate_uuid: "521378b341fb499cb4a56e5567691f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" is_pre_election: true
I20260926 02:19:46.872539 15185 raft_consensus.cc:2454] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521378b341fb499cb4a56e5567691f79 in term 0.
I20260926 02:19:46.872918 15468 raft_consensus.cc:2454] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521378b341fb499cb4a56e5567691f79 in term 0.
I20260926 02:19:46.873296 15255 leader_election.cc:302] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521378b341fb499cb4a56e5567691f79, 9607ad0bc7a146fcb0ca111e5b57f23b; no voters:
I20260926 02:19:46.873744 15539 raft_consensus.cc:2781] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:46.873932 15539 raft_consensus.cc:482] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:46.874049 15539 raft_consensus.cc:3037] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:46.876741 15539 raft_consensus.cc:504] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
I20260926 02:19:46.877355 15539 leader_election.cc:288] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [CANDIDATE]: Term 1 election: Requested vote from peers 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675), fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559)
I20260926 02:19:46.877991 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebab614c85484bb3bb37cb0b08731871" candidate_uuid: "521378b341fb499cb4a56e5567691f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b"
I20260926 02:19:46.878233 15185 raft_consensus.cc:3037] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:46.878198 15468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebab614c85484bb3bb37cb0b08731871" candidate_uuid: "521378b341fb499cb4a56e5567691f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd90f0fc17e48ad951a9e043ddcf879"
I20260926 02:19:46.878428 15468 raft_consensus.cc:3037] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:46.880679 15185 raft_consensus.cc:2454] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521378b341fb499cb4a56e5567691f79 in term 1.
I20260926 02:19:46.880832 15468 raft_consensus.cc:2454] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521378b341fb499cb4a56e5567691f79 in term 1.
I20260926 02:19:46.881173 15255 leader_election.cc:302] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521378b341fb499cb4a56e5567691f79, 9607ad0bc7a146fcb0ca111e5b57f23b; no voters:
I20260926 02:19:46.881596 15539 raft_consensus.cc:2781] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:46.882097 15539 raft_consensus.cc:686] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 1 LEADER]: Becoming Leader. State: Replica: 521378b341fb499cb4a56e5567691f79, State: Running, Role: LEADER
I20260926 02:19:46.882593 15539 consensus_queue.cc:237] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } }
W20260926 02:19:46.884824 15236 tablet.cc:2369] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:19:46.888015 15027 catalog_manager.cc:5495] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521378b341fb499cb4a56e5567691f79 (127.12.244.194). New cstate: current_term: 1 leader_uuid: "521378b341fb499cb4a56e5567691f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } health_report { overall_health: HEALTHY } } }
Sep 26 02:19:46 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389184, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:46.992863 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.977934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45759 (local address 127.0.0.1:48964)
0926 02:19:46.978696 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:46.978714 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:46.978903 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:46.979356 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:46.979360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:46.980081 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:46.980333 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.980342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.981473 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.981498 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.982460 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:46.982471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.983321 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.983326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:46.983443 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.984225 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:46.984242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:46.986509 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:46.990209 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:46.990225 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:46.990232 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:46.990716 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:46.991162 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:46.991167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:46.991172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:46.991309 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:46.991939 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:46.991949 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:46.992118 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 02:19:46.992252 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 02:19:46.992493 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":438,"threads_started":1}
I20260926 02:19:46.992956 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:46.977934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.12.244.254:45759)
0926 02:19:46.978142 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:46.978150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:46.978850 (+ 700us) server_negotiation.cc:407] Connection header received
0926 02:19:46.979051 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:46.979056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:46.979207 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:46.979302 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:46.980505 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.981340 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.982622 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:46.983147 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:46.983265 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:46.986724 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:46.986730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:46.986734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:46.986767 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:46.990008 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.990844 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.990849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.990852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.990995 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:46.991441 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:46.991445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:46.991447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:46.991747 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:46.991947 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:46.992360 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 02:19:46.992634 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":72}
I20260926 02:19:46.998888 15027 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:48964:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:19:47.000074 15027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:47.008111 15165 tablet_service.cc:1467] Processing CreateTablet for tablet 033e705fe9974878bd47f56460ae0b16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b3b5f1e8aac40d7a3a7cc391d683dc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:19:47.008351 15304 tablet_service.cc:1467] Processing CreateTablet for tablet 033e705fe9974878bd47f56460ae0b16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b3b5f1e8aac40d7a3a7cc391d683dc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:19:47.009132 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 033e705fe9974878bd47f56460ae0b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:47.009135 15165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 033e705fe9974878bd47f56460ae0b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:47.014588 15443 tablet_service.cc:1467] Processing CreateTablet for tablet 033e705fe9974878bd47f56460ae0b16 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6b3b5f1e8aac40d7a3a7cc391d683dc8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:19:47.015509 15443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 033e705fe9974878bd47f56460ae0b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:47.015570 15532 tablet_bootstrap.cc:492] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: Bootstrap starting.
I20260926 02:19:47.017243 15532 tablet_bootstrap.cc:654] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:47.019385 15533 tablet_bootstrap.cc:492] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: Bootstrap starting.
I20260926 02:19:47.020171 15532 tablet_bootstrap.cc:492] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: No bootstrap required, opened a new log
I20260926 02:19:47.020324 15532 ts_tablet_manager.cc:1397] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 02:19:47.020952 15533 tablet_bootstrap.cc:654] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:47.021127 15532 raft_consensus.cc:348] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.021368 15532 raft_consensus.cc:374] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:47.021452 15532 raft_consensus.cc:729] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9607ad0bc7a146fcb0ca111e5b57f23b, State: Initialized, Role: FOLLOWER
I20260926 02:19:47.021724 15532 consensus_queue.cc:260] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.022226 15538 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.022606 15538 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:47.023156 15532 ts_tablet_manager.cc:1428] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 02:19:47.023197 15534 tablet_bootstrap.cc:492] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: Bootstrap starting.
I20260926 02:19:47.024893 15534 tablet_bootstrap.cc:654] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:47.029214 15533 tablet_bootstrap.cc:492] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: No bootstrap required, opened a new log
I20260926 02:19:47.029384 15533 ts_tablet_manager.cc:1397] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260926 02:19:47.030118 15533 raft_consensus.cc:348] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.030243 15534 tablet_bootstrap.cc:492] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: No bootstrap required, opened a new log
I20260926 02:19:47.030325 15533 raft_consensus.cc:374] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:47.030423 15534 ts_tablet_manager.cc:1397] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 02:19:47.030478 15533 raft_consensus.cc:729] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521378b341fb499cb4a56e5567691f79, State: Initialized, Role: FOLLOWER
I20260926 02:19:47.030737 15533 consensus_queue.cc:260] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.031210 15539 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.031533 15539 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:47.031364 15534 raft_consensus.cc:348] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.031610 15534 raft_consensus.cc:374] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:47.031709 15534 raft_consensus.cc:729] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd90f0fc17e48ad951a9e043ddcf879, State: Initialized, Role: FOLLOWER
I20260926 02:19:47.032039 15533 ts_tablet_manager.cc:1428] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260926 02:19:47.032004 15534 consensus_queue.cc:260] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.032675 15534 ts_tablet_manager.cc:1428] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 02:19:47.032554 15540 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.032896 15540 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:47.213716 15538 raft_consensus.cc:482] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:47.213945 15538 raft_consensus.cc:504] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.215646 15538 leader_election.cc:288] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861), fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559)
I20260926 02:19:47.223071 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.215999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:41979 (local address 127.12.244.195:39559)
0926 02:19:47.216181 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.216187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.217199 (+ 1012us) server_negotiation.cc:407] Connection header received
0926 02:19:47.217373 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.217379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.217504 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.217617 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:47.218203 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.219062 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.221090 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.221936 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.222067 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.222162 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.222594 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.222752 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:19:47.223219 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.215998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:35297 (local address 127.12.244.194:45861)
0926 02:19:47.216582 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.216589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.216960 (+ 371us) server_negotiation.cc:407] Connection header received
0926 02:19:47.217176 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.217183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.217257 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.217371 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:47.218091 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.218863 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.221159 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.222053 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.222239 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.222332 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.222687 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.222857 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":305,"threads_started":1}
I20260926 02:19:47.223410 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.216231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:39559 (local address 127.12.244.193:41979)
0926 02:19:47.217065 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.217100 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.217224 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.217640 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.217643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.217661 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:47.218023 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.218037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.219256 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.219260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.220914 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.220932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.222164 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.222171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.222386 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.222450 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.222541 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.222691 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":50,"thread_start_us":294,"threads_started":1}
I20260926 02:19:47.223489 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.216070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.12.244.193:35297)
0926 02:19:47.216801 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.216827 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.217016 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.217393 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.217398 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.217422 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:47.217899 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.217920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.219064 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.219070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.220946 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.220959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.222302 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.222312 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.222488 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.222540 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.222642 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.222778 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":118,"thread_start_us":677,"threads_started":1}
I20260926 02:19:47.224148 15468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "033e705fe9974878bd47f56460ae0b16" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" is_pre_election: true
I20260926 02:19:47.224442 15468 raft_consensus.cc:2454] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 0.
I20260926 02:19:47.224411 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "033e705fe9974878bd47f56460ae0b16" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79" is_pre_election: true
I20260926 02:19:47.224881 15324 raft_consensus.cc:2454] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 0.
I20260926 02:19:47.224959 15115 leader_election.cc:302] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9607ad0bc7a146fcb0ca111e5b57f23b, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:47.225468 15538 raft_consensus.cc:2781] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:47.225631 15538 raft_consensus.cc:482] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:47.225744 15538 raft_consensus.cc:3037] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:47.228188 15538 raft_consensus.cc:504] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.228889 15538 leader_election.cc:288] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 election: Requested vote from peers 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861), fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559)
I20260926 02:19:47.229453 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "033e705fe9974878bd47f56460ae0b16" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79"
I20260926 02:19:47.229471 15468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "033e705fe9974878bd47f56460ae0b16" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd90f0fc17e48ad951a9e043ddcf879"
I20260926 02:19:47.229688 15324 raft_consensus.cc:3037] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:47.229725 15468 raft_consensus.cc:3037] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:47.233105 15468 raft_consensus.cc:2454] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 1.
I20260926 02:19:47.233124 15324 raft_consensus.cc:2454] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 1.
I20260926 02:19:47.233680 15115 leader_election.cc:302] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9607ad0bc7a146fcb0ca111e5b57f23b, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:47.234069 15538 raft_consensus.cc:2781] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:47.234536 15538 raft_consensus.cc:686] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 LEADER]: Becoming Leader. State: Replica: 9607ad0bc7a146fcb0ca111e5b57f23b, State: Running, Role: LEADER
I20260926 02:19:47.235085 15538 consensus_queue.cc:237] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.238463 15557 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: TxnStatusTablet state changed. Reason: New leader 9607ad0bc7a146fcb0ca111e5b57f23b. Latest consensus state: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.238938 15557 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:19:47.239764 15559 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:19:47.240054 15559 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:19:47.239915 15028 catalog_manager.cc:5495] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193). New cstate: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } health_report { overall_health: UNKNOWN } } }
I20260926 02:19:47.268359 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.261222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.0.0.1:48802)
0926 02:19:47.261501 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.261523 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.261707 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.262224 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.262229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.262245 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:47.262441 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.262453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.264131 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.264136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.265330 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.265344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.266390 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.266395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.266525 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.266587 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.267774 (+ 1187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.267804 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.267891 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.268039 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:19:47.268384 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.261300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.12.244.193:44675)
0926 02:19:47.261510 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.261515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.261633 (+ 118us) server_negotiation.cc:407] Connection header received
0926 02:19:47.261928 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.261934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.262033 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.262164 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:47.262626 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.263973 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.265520 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.266224 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.266357 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.266784 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.267625 (+ 841us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.267789 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.267934 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.268033 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 02:19:47.276902 15324 raft_consensus.cc:1261] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Refusing update from remote peer 9607ad0bc7a146fcb0ca111e5b57f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:47.277022 15468 raft_consensus.cc:1261] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Refusing update from remote peer 9607ad0bc7a146fcb0ca111e5b57f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:47.277944 15557 consensus_queue.cc:1035] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:47.278366 15538 consensus_queue.cc:1035] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:47.290271 15540 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: TxnStatusTablet state changed. Reason: New leader 9607ad0bc7a146fcb0ca111e5b57f23b. Latest consensus state: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.290606 15540 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:47.293010 15539 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: TxnStatusTablet state changed. Reason: New leader 9607ad0bc7a146fcb0ca111e5b57f23b. Latest consensus state: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.293368 15539 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:47.300688 15539 consensus_queue.cc:1035] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:47.308229 15540 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.308513 15540 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P fbd90f0fc17e48ad951a9e043ddcf879: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:47.316862 15567 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.317349 15567 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:19:47.319311 15563 mvcc.cc:205] Tried to move back new op lower bound from 7333434111075684352 to 7333434110918742016. Current Snapshot: MvccSnapshot[applied={T|T < 7333434111075684352}]
I20260926 02:19:47.321446 15539 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.321764 15539 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P 521378b341fb499cb4a56e5567691f79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:19:47.329571 15538 tablet_replica.cc:439] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } } }
I20260926 02:19:47.330118 15538 tablet_replica.cc:442] T 033e705fe9974878bd47f56460ae0b16 P 9607ad0bc7a146fcb0ca111e5b57f23b: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:19:47.333086 15543 consensus_queue.cc:1035] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:47.347334 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.338080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.12.244.194:45861)
0926 02:19:47.338260 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.338272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.341256 (+ 2984us) server_negotiation.cc:407] Connection header received
0926 02:19:47.341460 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.341467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.341574 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.341746 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:47.342224 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.343364 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.344760 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.345413 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.345545 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.345939 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.346710 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.346869 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.346917 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.347007 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:19:47.365772 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.358849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:41517 (local address 127.12.244.193:44675)
0926 02:19:47.359051 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.359058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.359684 (+ 626us) server_negotiation.cc:407] Connection header received
0926 02:19:47.359899 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.359906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.360016 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.360143 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:47.360686 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.361859 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.363497 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.364174 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.364327 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.364876 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.365019 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.365121 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.365343 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.365436 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:19:47.365819 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.358849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.12.244.194:41517)
0926 02:19:47.359570 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.359595 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.359734 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.360186 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.360190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.360201 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:47.360515 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.360524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.362023 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.362031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.363358 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.363372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.364396 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.364401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.364529 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.364629 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.365156 (+ 527us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.365168 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.365315 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.365456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":402,"threads_started":1}
I20260926 02:19:47.430531 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.422995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:39083 (local address 127.12.244.194:45861)
0926 02:19:47.423229 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.423235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.424016 (+ 781us) server_negotiation.cc:407] Connection header received
0926 02:19:47.424312 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.424320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.424440 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.424538 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:47.425334 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.426128 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.428300 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.428895 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.429065 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.429789 (+ 724us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.429942 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.430039 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.430154 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.430241 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:19:47.430716 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.423021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.12.244.193:39083)
0926 02:19:47.423909 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.423930 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.424133 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.424737 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.424742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.424756 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:47.425138 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.425149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.426267 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.426274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.428124 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.428142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.429234 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.429239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.429469 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.429576 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.430038 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.430050 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.430126 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.430298 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":416,"threads_started":1}
I20260926 02:19:47.480520 15028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:47.480845 15028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:47.485180 15028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Sending DeleteTablet for 3 replicas of tablet ebab614c85484bb3bb37cb0b08731871
I20260926 02:19:47.486642 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet ebab614c85484bb3bb37cb0b08731871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:47 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:50124
I20260926 02:19:47.492491 15443 tablet_service.cc:1514] Processing DeleteTablet for tablet ebab614c85484bb3bb37cb0b08731871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:47 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:57942
I20260926 02:19:47.495988 15028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:19:47.497641 15028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:47.500036 15586 tablet_replica.cc:330] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: stopping tablet replica
I20260926 02:19:47.500566 15586 raft_consensus.cc:2229] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:19:47.501309 15586 raft_consensus.cc:2258] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:47.506556 15586 ts_tablet_manager.cc:1905] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:47.508805 15304 tablet_service.cc:1467] Processing CreateTablet for tablet 00ba43c56b674b31b691fe832d9aed1b (DEFAULT_TABLE table=test-table [id=1c60c1562a4441b09afe2b7b75d062c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:47.509330 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ba43c56b674b31b691fe832d9aed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:47.516954 15164 tablet_service.cc:1467] Processing CreateTablet for tablet 00ba43c56b674b31b691fe832d9aed1b (DEFAULT_TABLE table=test-table [id=1c60c1562a4441b09afe2b7b75d062c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:47.517320 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 00ba43c56b674b31b691fe832d9aed1b (DEFAULT_TABLE table=test-table [id=1c60c1562a4441b09afe2b7b75d062c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:47.517539 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ba43c56b674b31b691fe832d9aed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:47.517897 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ba43c56b674b31b691fe832d9aed1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:47.525475 15165 tablet_service.cc:1514] Processing DeleteTablet for tablet ebab614c85484bb3bb37cb0b08731871 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:47 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:48790
I20260926 02:19:47.529242 15587 tablet_replica.cc:330] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: stopping tablet replica
I20260926 02:19:47.529299 15534 tablet_bootstrap.cc:492] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: Bootstrap starting.
I20260926 02:19:47.529729 15587 raft_consensus.cc:2229] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:47.530133 15587 raft_consensus.cc:2258] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:47.531842 15534 tablet_bootstrap.cc:654] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:47.532510 15587 ts_tablet_manager.cc:1905] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:47.536321 15589 tablet_replica.cc:330] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: stopping tablet replica
I20260926 02:19:47.536971 15589 raft_consensus.cc:2229] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:47.537321 15589 raft_consensus.cc:2258] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:47.538165 15588 tablet_bootstrap.cc:492] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: Bootstrap starting.
I20260926 02:19:47.540025 15586 ts_tablet_manager.cc:1918] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:47.540247 15586 log.cc:1196] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/ebab614c85484bb3bb37cb0b08731871
I20260926 02:19:47.540366 15588 tablet_bootstrap.cc:654] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:47.541108 15586 ts_tablet_manager.cc:1939] T ebab614c85484bb3bb37cb0b08731871 P 521378b341fb499cb4a56e5567691f79: Deleting consensus metadata
I20260926 02:19:47.542943 15014 catalog_manager.cc:4841] TS 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861): tablet ebab614c85484bb3bb37cb0b08731871 (table test-table [id=9d60adf985494cd681878ba3287407b2]) successfully deleted
I20260926 02:19:47.543495 15589 ts_tablet_manager.cc:1905] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:47.548413 15534 tablet_bootstrap.cc:492] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: No bootstrap required, opened a new log
I20260926 02:19:47.548609 15534 ts_tablet_manager.cc:1397] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260926 02:19:47.549398 15534 raft_consensus.cc:348] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.549655 15534 raft_consensus.cc:374] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:47.549728 15534 raft_consensus.cc:729] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd90f0fc17e48ad951a9e043ddcf879, State: Initialized, Role: FOLLOWER
I20260926 02:19:47.550069 15534 consensus_queue.cc:260] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.552088 15589 ts_tablet_manager.cc:1918] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:47.552335 15589 log.cc:1196] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/ebab614c85484bb3bb37cb0b08731871
I20260926 02:19:47.553804 15588 tablet_bootstrap.cc:492] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: No bootstrap required, opened a new log
I20260926 02:19:47.553912 15534 ts_tablet_manager.cc:1428] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 02:19:47.553989 15588 ts_tablet_manager.cc:1397] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20260926 02:19:47.554997 15588 raft_consensus.cc:348] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.555390 15588 raft_consensus.cc:374] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:47.555505 15588 raft_consensus.cc:729] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9607ad0bc7a146fcb0ca111e5b57f23b, State: Initialized, Role: FOLLOWER
I20260926 02:19:47.555826 15588 consensus_queue.cc:260] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.558634 15587 ts_tablet_manager.cc:1918] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:47.558861 15587 log.cc:1196] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/ebab614c85484bb3bb37cb0b08731871
I20260926 02:19:47.559468 15587 ts_tablet_manager.cc:1939] T ebab614c85484bb3bb37cb0b08731871 P fbd90f0fc17e48ad951a9e043ddcf879: Deleting consensus metadata
I20260926 02:19:47.559734 15588 ts_tablet_manager.cc:1428] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 02:19:47.560381 15592 tablet_bootstrap.cc:492] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: Bootstrap starting.
I20260926 02:19:47.561415 15012 catalog_manager.cc:4841] TS fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559): tablet ebab614c85484bb3bb37cb0b08731871 (table test-table [id=9d60adf985494cd681878ba3287407b2]) successfully deleted
I20260926 02:19:47.562307 15592 tablet_bootstrap.cc:654] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:47.566581 15589 ts_tablet_manager.cc:1939] T ebab614c85484bb3bb37cb0b08731871 P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting consensus metadata
I20260926 02:19:47.568635 15012 catalog_manager.cc:4841] TS 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675): tablet ebab614c85484bb3bb37cb0b08731871 (table test-table [id=9d60adf985494cd681878ba3287407b2]) successfully deleted
I20260926 02:19:47.573529 15592 tablet_bootstrap.cc:492] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: No bootstrap required, opened a new log
I20260926 02:19:47.573745 15592 ts_tablet_manager.cc:1397] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260926 02:19:47.574630 15592 raft_consensus.cc:348] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.574898 15592 raft_consensus.cc:374] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:47.574997 15592 raft_consensus.cc:729] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521378b341fb499cb4a56e5567691f79, State: Initialized, Role: FOLLOWER
I20260926 02:19:47.575322 15592 consensus_queue.cc:260] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.575918 15592 ts_tablet_manager.cc:1428] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:19:47.700131 15540 raft_consensus.cc:482] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:47.700423 15540 raft_consensus.cc:504] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.702407 15540 leader_election.cc:288] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675), 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861)
I20260926 02:19:47.722908 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.703307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:60809 (local address 127.12.244.193:44675)
0926 02:19:47.703529 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.703535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.713190 (+ 9655us) server_negotiation.cc:407] Connection header received
0926 02:19:47.713848 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.713855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.713937 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.714015 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:47.714872 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.716203 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.719424 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.721210 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.721422 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.721522 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.722387 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.722569 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:19:47.724287 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.702809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.12.244.195:60809)
0926 02:19:47.713021 (+ 10212us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.713053 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.713633 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.714240 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.714246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.714284 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:47.714670 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.714684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.716437 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.716443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.719135 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.719156 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.721897 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.721905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.722103 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.722149 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.723486 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.723664 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9997,"mutex_wait_us":1,"thread_start_us":10408,"threads_started":1}
I20260926 02:19:47.724417 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.703332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:49207 (local address 127.12.244.194:45861)
0926 02:19:47.703528 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.703535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.715740 (+ 12205us) server_negotiation.cc:407] Connection header received
0926 02:19:47.716207 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.716214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.716294 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.716356 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:19:47.717294 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.718434 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.721464 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.722943 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.723129 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.723202 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.723930 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.724103 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:19:47.725363 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.702922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.12.244.195:49207)
0926 02:19:47.715624 (+ 12702us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.715650 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.716025 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.716567 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.716573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.716598 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:19:47.717074 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.717086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.718639 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.718646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.721178 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.721202 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.723437 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.723444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.723649 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.723725 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.724728 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.724924 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12508,"thread_start_us":12826,"threads_started":1}
I20260926 02:19:47.725432 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ba43c56b674b31b691fe832d9aed1b" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" is_pre_election: true
I20260926 02:19:47.725724 15185 raft_consensus.cc:2454] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 0.
I20260926 02:19:47.726024 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ba43c56b674b31b691fe832d9aed1b" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79" is_pre_election: true
I20260926 02:19:47.726290 15324 raft_consensus.cc:2454] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 0.
I20260926 02:19:47.726713 15394 leader_election.cc:302] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9607ad0bc7a146fcb0ca111e5b57f23b, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:47.727690 15540 raft_consensus.cc:2781] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:47.727842 15540 raft_consensus.cc:482] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:47.727941 15540 raft_consensus.cc:3037] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:47.733170 15540 raft_consensus.cc:504] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.734721 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ba43c56b674b31b691fe832d9aed1b" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b"
I20260926 02:19:47.735231 15185 raft_consensus.cc:3037] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:47.738590 15185 raft_consensus.cc:2454] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 1.
I20260926 02:19:47.739317 15394 leader_election.cc:302] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9607ad0bc7a146fcb0ca111e5b57f23b, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:47.740561 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ba43c56b674b31b691fe832d9aed1b" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79"
I20260926 02:19:47.740814 15324 raft_consensus.cc:3037] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:47.744153 15324 raft_consensus.cc:2454] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 1.
I20260926 02:19:47.744851 15540 leader_election.cc:288] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 election: Requested vote from peers 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675), 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861)
I20260926 02:19:47.745150 15540 raft_consensus.cc:2781] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:47.745394 15540 raft_consensus.cc:686] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 LEADER]: Becoming Leader. State: Replica: fbd90f0fc17e48ad951a9e043ddcf879, State: Running, Role: LEADER
I20260926 02:19:47.745862 15540 consensus_queue.cc:237] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:47.758181 15027 catalog_manager.cc:5495] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } health_report { overall_health: HEALTHY } } }
I20260926 02:19:47.806330 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.794126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.12.244.195:39559)
0926 02:19:47.799685 (+ 5559us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.799690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.799718 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:19:47.799822 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.799826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.799928 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.800118 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:47.800674 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.801580 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.803553 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.804119 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.804332 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.804900 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.805521 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.805643 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.805804 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.805885 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5405,"thread_start_us":5238,"threads_started":1}
I20260926 02:19:47.825186 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.818758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:60493 (local address 127.12.244.193:44675)
0926 02:19:47.818982 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.818989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.819233 (+ 244us) server_negotiation.cc:407] Connection header received
0926 02:19:47.819417 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.819423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.819521 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.819628 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:47.820117 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.821257 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.822921 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.823659 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.823801 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.824329 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.824532 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.824640 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.824791 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.824883 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:19:47.825214 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.818375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.12.244.195:60493)
0926 02:19:47.819118 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.819141 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.819286 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.819655 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.819659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.819670 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:47.819966 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.819974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.821496 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.821500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.822749 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.822765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.823867 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.823873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.824069 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.824134 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.824652 (+ 518us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.824662 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.824738 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.824866 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":452,"threads_started":1}
I20260926 02:19:47.837065 15185 raft_consensus.cc:1261] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Refusing update from remote peer fbd90f0fc17e48ad951a9e043ddcf879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:47.837103 15324 raft_consensus.cc:1261] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Refusing update from remote peer fbd90f0fc17e48ad951a9e043ddcf879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:47.837836 15596 consensus_queue.cc:1035] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:47.838224 15540 consensus_queue.cc:1035] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:47.844310 15562 mvcc.cc:205] Tried to move back new op lower bound from 7333434113373728768 to 7333434113018576896. Current Snapshot: MvccSnapshot[applied={T|T < 7333434113373728768}]
I20260926 02:19:47.846326 15564 mvcc.cc:205] Tried to move back new op lower bound from 7333434113373728768 to 7333434113018576896. Current Snapshot: MvccSnapshot[applied={T|T < 7333434113373728768}]
I20260926 02:19:47.874323 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.868238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:60133 (local address 127.12.244.195:39559)
0926 02:19:47.868441 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:47.868446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:47.868461 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:19:47.868517 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:47.868521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:47.868647 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:47.868740 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:47.869327 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.870389 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.872056 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.872752 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.872901 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.873446 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.873591 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.873690 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:47.873902 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 02:19:47.874006 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:19:47.874356 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:47.867879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:39559 (local address 127.12.244.193:60133)
0926 02:19:47.868074 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:47.868114 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:47.868236 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:47.868997 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:47.869001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:47.869012 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:47.869172 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.869179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.870544 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.870552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.871875 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:47.871892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:47.872954 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:47.872961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:47.873200 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:47.873252 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:47.873752 (+ 500us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:47.873762 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:19:47.873872 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 02:19:47.874039 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 02:19:47.875591 15564 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:19:49.494971 15027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:49.495293 15027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:49.500036 15027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Sending DeleteTablet for 3 replicas of tablet 00ba43c56b674b31b691fe832d9aed1b
I20260926 02:19:49.501180 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ba43c56b674b31b691fe832d9aed1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:49 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:50124
I20260926 02:19:49.501221 15164 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ba43c56b674b31b691fe832d9aed1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:49 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:48790
I20260926 02:19:49.501996 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ba43c56b674b31b691fe832d9aed1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:49 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:57942
I20260926 02:19:49.512816 15626 tablet_replica.cc:330] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: stopping tablet replica
I20260926 02:19:49.513306 15626 raft_consensus.cc:2229] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:49.513949 15626 raft_consensus.cc:2258] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:49.514945 15627 tablet_replica.cc:330] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: stopping tablet replica
I20260926 02:19:49.515230 15627 raft_consensus.cc:2229] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:49.515605 15627 raft_consensus.cc:2258] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:49.516036 15628 tablet_replica.cc:330] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: stopping tablet replica
I20260926 02:19:49.516276 15628 raft_consensus.cc:2229] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:19:49.516908 15626 ts_tablet_manager.cc:1905] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:49.517131 15628 raft_consensus.cc:2258] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:49.518086 15627 ts_tablet_manager.cc:1905] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:49.519754 15628 ts_tablet_manager.cc:1905] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:49.524678 15626 ts_tablet_manager.cc:1918] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:19:49.524947 15626 log.cc:1196] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/00ba43c56b674b31b691fe832d9aed1b
I20260926 02:19:49.525862 15626 ts_tablet_manager.cc:1939] T 00ba43c56b674b31b691fe832d9aed1b P 521378b341fb499cb4a56e5567691f79: Deleting consensus metadata
I20260926 02:19:49.526000 15627 ts_tablet_manager.cc:1918] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:19:49.526288 15627 log.cc:1196] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/00ba43c56b674b31b691fe832d9aed1b
I20260926 02:19:49.527057 15627 ts_tablet_manager.cc:1939] T 00ba43c56b674b31b691fe832d9aed1b P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting consensus metadata
I20260926 02:19:49.527436 15014 catalog_manager.cc:4841] TS 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861): tablet 00ba43c56b674b31b691fe832d9aed1b (table test-table [id=1c60c1562a4441b09afe2b7b75d062c0]) successfully deleted
I20260926 02:19:49.528326 15628 ts_tablet_manager.cc:1918] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:19:49.528419 15012 catalog_manager.cc:4841] TS 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675): tablet 00ba43c56b674b31b691fe832d9aed1b (table test-table [id=1c60c1562a4441b09afe2b7b75d062c0]) successfully deleted
I20260926 02:19:49.528578 15628 log.cc:1196] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/00ba43c56b674b31b691fe832d9aed1b
I20260926 02:19:49.529094 15628 ts_tablet_manager.cc:1939] T 00ba43c56b674b31b691fe832d9aed1b P fbd90f0fc17e48ad951a9e043ddcf879: Deleting consensus metadata
I20260926 02:19:49.530045 15012 catalog_manager.cc:4841] TS fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559): tablet 00ba43c56b674b31b691fe832d9aed1b (table test-table [id=1c60c1562a4441b09afe2b7b75d062c0]) successfully deleted
Sep 26 02:19:49 dist-test-slave-qfmw krb5kdc[14983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:19:49 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:49.573598 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:49.555062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.12.244.254:45759)
0926 02:19:49.556049 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:49.556053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:49.556076 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:19:49.556130 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:49.556133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:49.556192 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:49.556288 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:49.557829 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.558614 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.560240 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.560705 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.560809 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:49.566278 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:49.566284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:49.566288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:49.566328 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:49.569799 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:49.570994 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:49.570999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:49.571001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:49.571240 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:49.571774 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:49.571779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:49.571781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:49.572182 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:49.572433 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:49.572842 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 02:19:49.573090 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":727,"threads_started":1}
I20260926 02:19:49.578087 15027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:19:49.578979 15027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:49.586418 15304 tablet_service.cc:1467] Processing CreateTablet for tablet 512f28e7bea448a581ee1c064f706546 (DEFAULT_TABLE table=test-table [id=11385da62b1f4e02859caacc5008717c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:49.586567 15164 tablet_service.cc:1467] Processing CreateTablet for tablet 512f28e7bea448a581ee1c064f706546 (DEFAULT_TABLE table=test-table [id=11385da62b1f4e02859caacc5008717c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:49.587016 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512f28e7bea448a581ee1c064f706546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:49.586978 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 512f28e7bea448a581ee1c064f706546 (DEFAULT_TABLE table=test-table [id=11385da62b1f4e02859caacc5008717c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:49.587167 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512f28e7bea448a581ee1c064f706546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:49.587496 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512f28e7bea448a581ee1c064f706546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:49.594821 15637 tablet_bootstrap.cc:492] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: Bootstrap starting.
I20260926 02:19:49.596715 15638 tablet_bootstrap.cc:492] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: Bootstrap starting.
I20260926 02:19:49.598004 15639 tablet_bootstrap.cc:492] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: Bootstrap starting.
I20260926 02:19:49.600031 15638 tablet_bootstrap.cc:654] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:49.600030 15637 tablet_bootstrap.cc:654] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:49.600973 15639 tablet_bootstrap.cc:654] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:49.605738 15637 tablet_bootstrap.cc:492] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: No bootstrap required, opened a new log
I20260926 02:19:49.606043 15637 ts_tablet_manager.cc:1397] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260926 02:19:49.606374 15639 tablet_bootstrap.cc:492] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: No bootstrap required, opened a new log
I20260926 02:19:49.606618 15639 ts_tablet_manager.cc:1397] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260926 02:19:49.607268 15638 tablet_bootstrap.cc:492] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: No bootstrap required, opened a new log
I20260926 02:19:49.607487 15638 ts_tablet_manager.cc:1397] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260926 02:19:49.607544 15637 raft_consensus.cc:348] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.607959 15637 raft_consensus.cc:374] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:49.607887 15639 raft_consensus.cc:348] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.608194 15637 raft_consensus.cc:729] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9607ad0bc7a146fcb0ca111e5b57f23b, State: Initialized, Role: FOLLOWER
I20260926 02:19:49.608220 15639 raft_consensus.cc:374] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:49.608397 15639 raft_consensus.cc:729] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd90f0fc17e48ad951a9e043ddcf879, State: Initialized, Role: FOLLOWER
I20260926 02:19:49.608561 15637 consensus_queue.cc:260] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.608769 15639 consensus_queue.cc:260] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.608862 15638 raft_consensus.cc:348] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.609189 15638 raft_consensus.cc:374] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:49.609395 15638 raft_consensus.cc:729] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521378b341fb499cb4a56e5567691f79, State: Initialized, Role: FOLLOWER
I20260926 02:19:49.609468 15637 ts_tablet_manager.cc:1428] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:49.609760 15638 consensus_queue.cc:260] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.611222 15639 ts_tablet_manager.cc:1428] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 02:19:49.611392 15638 ts_tablet_manager.cc:1428] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260926 02:19:49.780768 15538 raft_consensus.cc:482] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:49.781002 15538 raft_consensus.cc:504] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.781703 15538 leader_election.cc:288] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861), fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559)
I20260926 02:19:49.782549 15468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f28e7bea448a581ee1c064f706546" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" is_pre_election: true
I20260926 02:19:49.782604 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f28e7bea448a581ee1c064f706546" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79" is_pre_election: true
I20260926 02:19:49.782938 15468 raft_consensus.cc:2454] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 0.
I20260926 02:19:49.783023 15324 raft_consensus.cc:2454] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 0.
I20260926 02:19:49.783756 15115 leader_election.cc:302] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 521378b341fb499cb4a56e5567691f79, 9607ad0bc7a146fcb0ca111e5b57f23b, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:49.784200 15538 raft_consensus.cc:2781] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:49.784426 15538 raft_consensus.cc:482] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:49.784593 15538 raft_consensus.cc:3037] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:49.787669 15538 raft_consensus.cc:504] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.788519 15538 leader_election.cc:288] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 election: Requested vote from peers 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861), fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559)
I20260926 02:19:49.789294 15468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f28e7bea448a581ee1c064f706546" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd90f0fc17e48ad951a9e043ddcf879"
I20260926 02:19:49.789364 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512f28e7bea448a581ee1c064f706546" candidate_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79"
I20260926 02:19:49.789698 15468 raft_consensus.cc:3037] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:49.789716 15324 raft_consensus.cc:3037] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:49.793969 15468 raft_consensus.cc:2454] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 1.
I20260926 02:19:49.794016 15324 raft_consensus.cc:2454] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9607ad0bc7a146fcb0ca111e5b57f23b in term 1.
I20260926 02:19:49.794633 15115 leader_election.cc:302] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 521378b341fb499cb4a56e5567691f79, 9607ad0bc7a146fcb0ca111e5b57f23b, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:49.795122 15538 raft_consensus.cc:2781] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:49.795348 15538 raft_consensus.cc:686] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 LEADER]: Becoming Leader. State: Replica: 9607ad0bc7a146fcb0ca111e5b57f23b, State: Running, Role: LEADER
I20260926 02:19:49.795768 15538 consensus_queue.cc:237] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:49.799230 15027 catalog_manager.cc:5495] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193). New cstate: current_term: 1 leader_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } health_report { overall_health: UNKNOWN } } }
I20260926 02:19:49.855597 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:49.847803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.12.244.193:44675)
0926 02:19:49.848891 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:49.848897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:49.848939 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:19:49.849036 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:49.849040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:49.849134 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:49.849258 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:49.849805 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.850976 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.853107 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.853707 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.853860 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:49.854524 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:49.854662 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:49.854736 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:49.854998 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 02:19:49.855087 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":593,"threads_started":1}
I20260926 02:19:49.873953 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:49.866803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:47281 (local address 127.12.244.193:44675)
0926 02:19:49.867050 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:49.867059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:49.867747 (+ 688us) server_negotiation.cc:407] Connection header received
0926 02:19:49.867973 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:49.867977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:49.868036 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:49.868162 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:49.868859 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.869861 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.871858 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.872530 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.872648 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:49.873278 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:49.873383 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:49.873458 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:49.873606 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 02:19:49.873670 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 02:19:49.874217 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:49.866544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.12.244.193:47281)
0926 02:19:49.867635 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:49.867653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:49.867839 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:49.868184 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:49.868187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:49.868197 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:19:49.868634 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:49.868643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.869983 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.869987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:49.871730 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:49.871742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:49.872794 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:49.872800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:49.873022 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:49.873062 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:19:49.873488 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:19:49.873500 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:19:49.873573 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:19:49.873684 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":753,"threads_started":1}
I20260926 02:19:49.883975 15468 raft_consensus.cc:1261] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Refusing update from remote peer 9607ad0bc7a146fcb0ca111e5b57f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:49.883975 15324 raft_consensus.cc:1261] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Refusing update from remote peer 9607ad0bc7a146fcb0ca111e5b57f23b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:49.884699 15608 consensus_queue.cc:1035] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:49.885283 15613 consensus_queue.cc:1035] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:19:49.952085 15027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:49.952474 15027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:49.957232 15027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Sending DeleteTablet for 3 replicas of tablet 512f28e7bea448a581ee1c064f706546
I20260926 02:19:49.958436 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 512f28e7bea448a581ee1c064f706546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:49 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:50124
I20260926 02:19:49.958509 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 512f28e7bea448a581ee1c064f706546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:49 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:57942
I20260926 02:19:49.958832 15628 tablet_replica.cc:330] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: stopping tablet replica
I20260926 02:19:49.958850 15164 tablet_service.cc:1514] Processing DeleteTablet for tablet 512f28e7bea448a581ee1c064f706546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:49 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:48790
I20260926 02:19:49.959335 15627 tablet_replica.cc:330] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: stopping tablet replica
I20260926 02:19:49.959319 15626 tablet_replica.cc:330] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: stopping tablet replica
I20260926 02:19:49.959628 15626 raft_consensus.cc:2229] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:49.959645 15627 raft_consensus.cc:2229] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:19:49.959899 15626 raft_consensus.cc:2258] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:49.959066 15628 raft_consensus.cc:2229] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:49.960392 15627 raft_consensus.cc:2258] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:49.960523 15628 raft_consensus.cc:2258] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:49.961696 15626 ts_tablet_manager.cc:1905] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:49.961593 15027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:19:49.961946 15628 ts_tablet_manager.cc:1905] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 02:19:49.962708 15027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:19:49.965304 15627 ts_tablet_manager.cc:1905] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:49.970078 15626 ts_tablet_manager.cc:1918] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:49.973425 15626 log.cc:1196] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/512f28e7bea448a581ee1c064f706546
I20260926 02:19:49.974174 15628 ts_tablet_manager.cc:1918] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:49.974444 15628 log.cc:1196] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/512f28e7bea448a581ee1c064f706546
I20260926 02:19:49.974504 15626 ts_tablet_manager.cc:1939] T 512f28e7bea448a581ee1c064f706546 P 521378b341fb499cb4a56e5567691f79: Deleting consensus metadata
I20260926 02:19:49.976904 15304 tablet_service.cc:1467] Processing CreateTablet for tablet 97e8ac227aa14198b82138ef44bd89be (DEFAULT_TABLE table=test-table [id=7a61c6dd8ddb41af92f607f27ef64d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:49.977393 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e8ac227aa14198b82138ef44bd89be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:49.977936 15628 ts_tablet_manager.cc:1939] T 512f28e7bea448a581ee1c064f706546 P fbd90f0fc17e48ad951a9e043ddcf879: Deleting consensus metadata
I20260926 02:19:49.978060 15164 tablet_service.cc:1467] Processing CreateTablet for tablet 97e8ac227aa14198b82138ef44bd89be (DEFAULT_TABLE table=test-table [id=7a61c6dd8ddb41af92f607f27ef64d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:49.978709 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e8ac227aa14198b82138ef44bd89be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:49.981297 15627 ts_tablet_manager.cc:1918] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:19:49.981488 15012 catalog_manager.cc:4841] TS fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559): tablet 512f28e7bea448a581ee1c064f706546 (table test-table [id=11385da62b1f4e02859caacc5008717c]) successfully deleted
I20260926 02:19:49.981588 15627 log.cc:1196] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/512f28e7bea448a581ee1c064f706546
I20260926 02:19:49.981544 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 97e8ac227aa14198b82138ef44bd89be (DEFAULT_TABLE table=test-table [id=7a61c6dd8ddb41af92f607f27ef64d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:19:49.982087 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97e8ac227aa14198b82138ef44bd89be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:49.982198 15627 ts_tablet_manager.cc:1939] T 512f28e7bea448a581ee1c064f706546 P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting consensus metadata
I20260926 02:19:49.983516 15012 catalog_manager.cc:4841] TS 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675): tablet 512f28e7bea448a581ee1c064f706546 (table test-table [id=11385da62b1f4e02859caacc5008717c]) successfully deleted
I20260926 02:19:49.991029 15639 tablet_bootstrap.cc:492] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: Bootstrap starting.
I20260926 02:19:49.991559 15637 tablet_bootstrap.cc:492] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b: Bootstrap starting.
I20260926 02:19:49.994555 15638 tablet_bootstrap.cc:492] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: Bootstrap starting.
I20260926 02:19:49.994978 15639 tablet_bootstrap.cc:654] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:49.995241 15014 catalog_manager.cc:4841] TS 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861): tablet 512f28e7bea448a581ee1c064f706546 (table test-table [id=11385da62b1f4e02859caacc5008717c]) successfully deleted
I20260926 02:19:49.997547 15638 tablet_bootstrap.cc:654] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:49.997706 15637 tablet_bootstrap.cc:654] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:50.002784 15639 tablet_bootstrap.cc:492] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: No bootstrap required, opened a new log
I20260926 02:19:50.003082 15639 ts_tablet_manager.cc:1397] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260926 02:19:50.004051 15637 tablet_bootstrap.cc:492] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b: No bootstrap required, opened a new log
I20260926 02:19:50.004292 15637 ts_tablet_manager.cc:1397] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20260926 02:19:50.004413 15639 raft_consensus.cc:348] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.004887 15639 raft_consensus.cc:374] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:50.005017 15639 raft_consensus.cc:729] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd90f0fc17e48ad951a9e043ddcf879, State: Initialized, Role: FOLLOWER
I20260926 02:19:50.005609 15637 raft_consensus.cc:348] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.005662 15639 consensus_queue.cc:260] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.005991 15637 raft_consensus.cc:374] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:50.006153 15637 raft_consensus.cc:729] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9607ad0bc7a146fcb0ca111e5b57f23b, State: Initialized, Role: FOLLOWER
I20260926 02:19:50.006727 15639 ts_tablet_manager.cc:1428] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:50.007010 15637 consensus_queue.cc:260] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.007817 15637 ts_tablet_manager.cc:1428] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:50.008715 15638 tablet_bootstrap.cc:492] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: No bootstrap required, opened a new log
I20260926 02:19:50.008989 15638 ts_tablet_manager.cc:1397] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260926 02:19:50.010162 15638 raft_consensus.cc:348] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.010807 15638 raft_consensus.cc:374] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:50.010932 15638 raft_consensus.cc:729] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521378b341fb499cb4a56e5567691f79, State: Initialized, Role: FOLLOWER
I20260926 02:19:50.011323 15638 consensus_queue.cc:260] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.012094 15638 ts_tablet_manager.cc:1428] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:50.034839 15618 raft_consensus.cc:482] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:19:50.035213 15618 raft_consensus.cc:504] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.036216 15618 leader_election.cc:288] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675), 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861)
I20260926 02:19:50.036890 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e8ac227aa14198b82138ef44bd89be" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" is_pre_election: true
I20260926 02:19:50.037108 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e8ac227aa14198b82138ef44bd89be" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79" is_pre_election: true
I20260926 02:19:50.037425 15185 raft_consensus.cc:2454] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 0.
I20260926 02:19:50.037431 15324 raft_consensus.cc:2454] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 0.
I20260926 02:19:50.038138 15394 leader_election.cc:302] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9607ad0bc7a146fcb0ca111e5b57f23b, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:50.038596 15618 raft_consensus.cc:2781] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:19:50.038839 15618 raft_consensus.cc:482] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:19:50.039023 15618 raft_consensus.cc:3037] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:50.042640 15618 raft_consensus.cc:504] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.043478 15618 leader_election.cc:288] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 election: Requested vote from peers 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675), 521378b341fb499cb4a56e5567691f79 (127.12.244.194:45861)
I20260926 02:19:50.044296 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e8ac227aa14198b82138ef44bd89be" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521378b341fb499cb4a56e5567691f79"
I20260926 02:19:50.044397 15185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97e8ac227aa14198b82138ef44bd89be" candidate_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b"
I20260926 02:19:50.044854 15324 raft_consensus.cc:3037] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:50.044904 15185 raft_consensus.cc:3037] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:50.049302 15324 raft_consensus.cc:2454] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 1.
I20260926 02:19:50.049429 15185 raft_consensus.cc:2454] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd90f0fc17e48ad951a9e043ddcf879 in term 1.
I20260926 02:19:50.050071 15396 leader_election.cc:302] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 521378b341fb499cb4a56e5567691f79, fbd90f0fc17e48ad951a9e043ddcf879; no voters:
I20260926 02:19:50.050719 15618 raft_consensus.cc:2781] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:50.057818 15618 raft_consensus.cc:686] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 LEADER]: Becoming Leader. State: Replica: fbd90f0fc17e48ad951a9e043ddcf879, State: Running, Role: LEADER
I20260926 02:19:50.058620 15618 consensus_queue.cc:237] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } }
I20260926 02:19:50.062326 15027 catalog_manager.cc:5495] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd90f0fc17e48ad951a9e043ddcf879" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 39559 } health_report { overall_health: HEALTHY } } }
I20260926 02:19:50.093541 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:50.085324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.12.244.195:39559)
0926 02:19:50.087242 (+ 1918us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:50.087247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:50.087281 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:19:50.087341 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:50.087345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:50.087410 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:50.087542 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:19:50.088327 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:50.089340 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:50.090957 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:50.091652 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:50.091803 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:50.092506 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:19:50.092636 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:19:50.092747 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:50.092955 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 02:19:50.093046 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1635,"thread_start_us":1302,"threads_started":1}
I20260926 02:19:50.107702 15185 raft_consensus.cc:1261] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Refusing update from remote peer fbd90f0fc17e48ad951a9e043ddcf879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:50.108513 15324 raft_consensus.cc:1261] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Refusing update from remote peer fbd90f0fc17e48ad951a9e043ddcf879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:19:50.108546 15658 consensus_queue.cc:1035] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9607ad0bc7a146fcb0ca111e5b57f23b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 02:19:50.109233 15618 consensus_queue.cc:1035] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521378b341fb499cb4a56e5567691f79" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:19:50.138736 15609 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:19:51.747987 15027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:51.748219 15027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:19:51.754269 15027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c275afc36a7542f5b2480ede5e930b3a: Sending DeleteTablet for 3 replicas of tablet 97e8ac227aa14198b82138ef44bd89be
I20260926 02:19:51.755349 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 97e8ac227aa14198b82138ef44bd89be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:51 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:50124
I20260926 02:19:51.755364 15164 tablet_service.cc:1514] Processing DeleteTablet for tablet 97e8ac227aa14198b82138ef44bd89be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:51 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:48790
I20260926 02:19:51.755445 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 97e8ac227aa14198b82138ef44bd89be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:19:51 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:57942
I20260926 02:19:51.756345 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15099
I20260926 02:19:51.769601 15687 tablet_replica.cc:330] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: stopping tablet replica
I20260926 02:19:51.770143 15687 raft_consensus.cc:2229] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:51.770553 15687 raft_consensus.cc:2258] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:51.772737 15687 ts_tablet_manager.cc:1905] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:51.776587 15688 tablet_replica.cc:330] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b: stopping tablet replica
I20260926 02:19:51.776935 15688 raft_consensus.cc:2229] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:19:51.777288 15688 raft_consensus.cc:2258] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:51.777591 15689 tablet_replica.cc:330] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: stopping tablet replica
I20260926 02:19:51.777805 15689 raft_consensus.cc:2229] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:19:51.778237 15689 raft_consensus.cc:2258] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:19:51.778359 15688 ts_tablet_manager.cc:1905] T 97e8ac227aa14198b82138ef44bd89be P 9607ad0bc7a146fcb0ca111e5b57f23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:19:51.779284 15689 ts_tablet_manager.cc:1905] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:19:51 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:19:51.795652 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:51.758323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.12.244.193:44675)
0926 02:19:51.765472 (+ 7149us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:51.765480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:51.765524 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:19:51.765608 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:51.765617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:51.765728 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:51.765874 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:51.768562 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:51.770082 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:51.772882 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:51.774262 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:51.774447 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:51.787768 (+ 13321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:51.787786 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:51.787804 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:51.787859 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:51.791719 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:51.792833 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:51.792842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:51.792844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:51.793167 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:51.793623 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:51.793626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:51.793628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:51.794162 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:51.794446 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:51.794828 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 02:19:51.795068 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6937,"thread_start_us":6683,"threads_started":1}
W20260926 02:19:51.796478 15229 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47252
W20260926 02:19:51.797304 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15099: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:51.797623 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 15099
I20260926 02:19:51.879230 15687 ts_tablet_manager.cc:1918] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:19:51.879213 15689 ts_tablet_manager.cc:1918] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:19:51.879438 15689 log.cc:1196] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/97e8ac227aa14198b82138ef44bd89be
I20260926 02:19:51.879485 15687 log.cc:1196] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/97e8ac227aa14198b82138ef44bd89be
I20260926 02:19:51.880053 15687 ts_tablet_manager.cc:1939] T 97e8ac227aa14198b82138ef44bd89be P 521378b341fb499cb4a56e5567691f79: Deleting consensus metadata
I20260926 02:19:51.880065 15689 ts_tablet_manager.cc:1939] T 97e8ac227aa14198b82138ef44bd89be P fbd90f0fc17e48ad951a9e043ddcf879: Deleting consensus metadata
I20260926 02:19:51.899971 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15239
W20260926 02:19:51.900128 15012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.244.193:44675 (error 108)
I20260926 02:19:51.903231 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:51.902008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.0.0.1:47260)
0926 02:19:51.902722 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:51.902884 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":474,"threads_started":1}
W20260926 02:19:51.903481 15012 catalog_manager.cc:4584] TS 9607ad0bc7a146fcb0ca111e5b57f23b (127.12.244.193:44675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97e8ac227aa14198b82138ef44bd89be: Network error: Client connection negotiation failed: client connection to 127.12.244.193:44675: connect: Connection refused (error 111)
Sep 26 02:19:51 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:19:51.918702 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:51.900824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.12.244.194:45861)
0926 02:19:51.901789 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:51.901796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:51.901817 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:19:51.901882 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:51.901887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:51.902076 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:51.902194 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:51.903952 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:51.904896 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:51.906693 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:51.907211 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:51.907353 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:51.911949 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:51.911966 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:51.911984 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:51.912029 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:51.915428 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:51.916312 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:51.916321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:51.916324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:51.916510 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:51.916995 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:51.917003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:51.917006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:51.917399 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:51.917756 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:51.918078 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 02:19:51.918333 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":818,"thread_start_us":695,"threads_started":1}
W20260926 02:19:51.919658 15369 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44478
W20260926 02:19:51.920410 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15239: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:51.920589 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 15239
I20260926 02:19:51.950866 15012 catalog_manager.cc:4841] TS fbd90f0fc17e48ad951a9e043ddcf879 (127.12.244.195:39559): tablet 97e8ac227aa14198b82138ef44bd89be (table test-table [id=7a61c6dd8ddb41af92f607f27ef64d63]) successfully deleted
I20260926 02:19:51.957486 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:51.956986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.0.0.1:47274)
0926 02:19:51.957218 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:51.957316 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 02:19:51.958699 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:51.958283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.0.0.1:47288)
0926 02:19:51.958447 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:51.958550 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 02:19:51.972664 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15378
I20260926 02:19:51.974156 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:51.973436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.0.0.1:44494)
0926 02:19:51.973793 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:51.973924 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.194:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
Sep 26 02:19:51 dist-test-slave-qfmw krb5kdc[14983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:19:51.991209 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:51.973499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.12.244.195:39559)
0926 02:19:51.974417 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:51.974425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:51.974448 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:19:51.974515 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:51.974523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:51.974619 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:51.974738 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:51.976293 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:51.977459 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:51.978928 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:51.979449 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:51.979560 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:51.984478 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:51.984495 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:51.984513 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:51.984557 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:51.987790 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:51.988708 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:51.988717 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:51.988719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:51.988944 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:51.989389 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:51.989393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:51.989395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:51.989791 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:51.990094 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:51.990421 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:19:51.990633 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":628,"threads_started":1}
W20260926 02:19:51.991937 15508 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47180
W20260926 02:19:51.992609 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15378: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:19:51.992851 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 15378
I20260926 02:19:52.007908 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:52.007316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.0.0.1:44498)
0926 02:19:52.007551 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:52.007650 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.194:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:19:52.009306 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:52.008819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.0.0.1:44500)
0926 02:19:52.009047 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:52.009142 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.194:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 02:19:52.017218 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 14996
I20260926 02:19:52.018844 15068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:52.141381 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:52.018744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.0.0.1:47290)
0926 02:19:52.019223 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:52.019319 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260926 02:19:52.142189 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:52.140971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45861 (local address 127.0.0.1:44514)
0926 02:19:52.141823 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:52.141953 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.194:45861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"threads_started":1}
I20260926 02:19:52.142767 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:52.142335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44675 (local address 127.0.0.1:47300)
0926 02:19:52.142519 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:52.142602 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:44675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 02:19:52.142818 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 14996
2026-09-26T02:19:52Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9231 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:54 dist-test-slave-qfmw krb5kdc[15700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:54 dist-test-slave-qfmw krb5kdc[15700](info): set up 2 sockets
Sep 26 02:19:54 dist-test-slave-qfmw krb5kdc[15700](info): commencing operation
krb5kdc: starting...
W20260926 02:19:55.318370 13267 mini_kdc.cc:121] Time spent starting KDC: real 3.131s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 02:19:55.469650 13267 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.151s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 02:19:55.612977 13267 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.143s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:19:55 dist-test-slave-qfmw krb5kdc[15700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:19:55.759459 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33117
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:56.095167 15713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:56.095553 15713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:56.095722 15713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:56.103235 15713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:56.103394 15713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:56.103646 15713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:56.103799 15713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:56.103892 15713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:56.114691 15713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:33117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:56.116272 15713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:56.117794 15713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:56.131513 15721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:56.131619 15719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:56.131690 15718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:56.132840 15713 server_base.cc:1034] running on GCE node
I20260926 02:19:56.133755 15713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:56.135522 15713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:56.136750 15713 hybrid_clock.cc:648] HybridClock initialized: now 1790389196136682 us; error 65 us; skew 500 ppm
I20260926 02:19:56.137158 15713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:56 dist-test-slave-qfmw krb5kdc[15700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:56.140720 15713 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:19:56.142390 15713 webserver.cc:460] Webserver started at http://127.12.244.254:45027/ using document root <none> and password file <none>
I20260926 02:19:56.143214 15713 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:56.143373 15713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:56.143703 15713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:56.146981 15713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "998427cc61524ad0a452dc2d136db341"
format_stamp: "Formatted at 2026-09-26 02:19:56 on dist-test-slave-qfmw"
server_key: "d369ba1f64ded3430276bcc87b325115"
server_key_iv: "c85559ee5ae7a25f7620d1fff91dfbb6"
server_key_version: "encryptionkey@0"
I20260926 02:19:56.148002 15713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "998427cc61524ad0a452dc2d136db341"
format_stamp: "Formatted at 2026-09-26 02:19:56 on dist-test-slave-qfmw"
server_key: "d369ba1f64ded3430276bcc87b325115"
server_key_iv: "c85559ee5ae7a25f7620d1fff91dfbb6"
server_key_version: "encryptionkey@0"
I20260926 02:19:56.154034 15713 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:19:56.158085 15728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:56.159731 15713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:19:56.159919 15713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "998427cc61524ad0a452dc2d136db341"
format_stamp: "Formatted at 2026-09-26 02:19:56 on dist-test-slave-qfmw"
server_key: "d369ba1f64ded3430276bcc87b325115"
server_key_iv: "c85559ee5ae7a25f7620d1fff91dfbb6"
server_key_version: "encryptionkey@0"
I20260926 02:19:56.160123 15713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:56.180186 15713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:56.184891 15713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:56.185329 15713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:56.205137 15713 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:33117
I20260926 02:19:56.205150 15779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:33117 every 8 connection(s)
I20260926 02:19:56.206707 15713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:56.209604 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 15713
I20260926 02:19:56.209839 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:19:56.210618 13267 external_mini_cluster.cc:1428] Setting key f94390354ef4f969285c96e251187b3f
I20260926 02:19:56.211640 15780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:56.224565 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341: Bootstrap starting.
Sep 26 02:19:56 dist-test-slave-qfmw krb5kdc[15700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:56.228530 15780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:56.229743 15780 log.cc:824] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:56.233546 15780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341: No bootstrap required, opened a new log
I20260926 02:19:56.234493 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:56.212886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35564 (local address 127.12.244.254:33117)
0926 02:19:56.213709 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:56.213723 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:56.213763 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:19:56.215417 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:56.215442 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:56.215913 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:56.216363 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:56.217891 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:56.219330 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:56.220622 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:56.221357 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:56.221591 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:56.226161 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:56.226169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:56.226188 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:56.226240 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:56.229594 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:56.230734 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:56.230744 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:56.230752 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:56.231041 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:56.231575 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:56.231579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:56.231581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:56.232127 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:56.232354 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:56.232856 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 02:19:56.233298 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":369,"threads_started":1}
I20260926 02:19:56.241731 15780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998427cc61524ad0a452dc2d136db341" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33117 } }
I20260926 02:19:56.242182 15780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:56.242318 15780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998427cc61524ad0a452dc2d136db341, State: Initialized, Role: FOLLOWER
I20260926 02:19:56.242959 15780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998427cc61524ad0a452dc2d136db341" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33117 } }
I20260926 02:19:56.243279 15780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:56.243422 15780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:56.243602 15780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:56.245874 15780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998427cc61524ad0a452dc2d136db341" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33117 } }
I20260926 02:19:56.246433 15780 leader_election.cc:302] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 998427cc61524ad0a452dc2d136db341; no voters:
I20260926 02:19:56.247140 15780 leader_election.cc:288] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:56.247318 15785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:56.248034 15785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [term 1 LEADER]: Becoming Leader. State: Replica: 998427cc61524ad0a452dc2d136db341, State: Running, Role: LEADER
I20260926 02:19:56.248658 15785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998427cc61524ad0a452dc2d136db341" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33117 } }
I20260926 02:19:56.249428 15780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:56.252043 15786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "998427cc61524ad0a452dc2d136db341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998427cc61524ad0a452dc2d136db341" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33117 } } }
I20260926 02:19:56.252640 15786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:56.252045 15787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 998427cc61524ad0a452dc2d136db341. Latest consensus state: current_term: 1 leader_uuid: "998427cc61524ad0a452dc2d136db341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "998427cc61524ad0a452dc2d136db341" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33117 } } }
I20260926 02:19:56.253459 15787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:56.254196 15791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:56.262034 15791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:56.272979 15791 catalog_manager.cc:1347] Generated new cluster ID: c8955fdc1ca24401bee6bc25d6a91c30
I20260926 02:19:56.273188 15791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:19:56.300735 15791 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:19:56.302146 15791 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:19:56.310781 15791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 998427cc61524ad0a452dc2d136db341: Generated new TSK 0
I20260926 02:19:56.312052 15791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:19:56.367380 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33117
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:56.717609 15806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:56.717918 15806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:56.718014 15806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:56.725740 15806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:56.725862 15806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:56.725976 15806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:19:56.736482 15806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:56.737830 15806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:56.739202 15806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:56.752977 15811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:56.752923 15812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:56.753556 15814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:56.754155 15806 server_base.cc:1034] running on GCE node
I20260926 02:19:56.754844 15806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:56.755934 15806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:56.757208 15806 hybrid_clock.cc:648] HybridClock initialized: now 1790389196757148 us; error 44 us; skew 500 ppm
I20260926 02:19:56.757613 15806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:56 dist-test-slave-qfmw krb5kdc[15700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:56.761005 15806 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:19:56.762743 15806 webserver.cc:460] Webserver started at http://127.12.244.193:41467/ using document root <none> and password file <none>
I20260926 02:19:56.763618 15806 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:56.763788 15806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:56.764161 15806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:56.767416 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "095a5d1f816640b59ea06c5b05a672d3"
format_stamp: "Formatted at 2026-09-26 02:19:56 on dist-test-slave-qfmw"
server_key: "8613b5749b3114236a5b1ad54a008a3d"
server_key_iv: "003a173382e45d0ca797156f22a42508"
server_key_version: "encryptionkey@0"
I20260926 02:19:56.768267 15806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "095a5d1f816640b59ea06c5b05a672d3"
format_stamp: "Formatted at 2026-09-26 02:19:56 on dist-test-slave-qfmw"
server_key: "8613b5749b3114236a5b1ad54a008a3d"
server_key_iv: "003a173382e45d0ca797156f22a42508"
server_key_version: "encryptionkey@0"
I20260926 02:19:56.773802 15806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:19:56.777621 15821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:56.779268 15806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:19:56.779465 15806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "095a5d1f816640b59ea06c5b05a672d3"
format_stamp: "Formatted at 2026-09-26 02:19:56 on dist-test-slave-qfmw"
server_key: "8613b5749b3114236a5b1ad54a008a3d"
server_key_iv: "003a173382e45d0ca797156f22a42508"
server_key_version: "encryptionkey@0"
I20260926 02:19:56.779680 15806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:56.797164 15806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:56.801451 15806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:56.801770 15806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:56.802769 15806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:56.804237 15806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:56.804316 15806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:56.804448 15806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:56.804514 15806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:56.841185 15806 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:41925
I20260926 02:19:56.841228 15933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:41925 every 8 connection(s)
I20260926 02:19:56.842733 15806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:19:56.847656 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 15806
I20260926 02:19:56.847899 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:19:56.848750 13267 external_mini_cluster.cc:1428] Setting key ac399f5eb11b3e09407130ff602aa017
Sep 26 02:19:56 dist-test-slave-qfmw krb5kdc[15700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:56.868299 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:56.845626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:39521 (local address 127.12.244.254:33117)
0926 02:19:56.846137 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:56.846141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:56.847315 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 02:19:56.850083 (+ 2768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:56.850088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:56.850187 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:56.850318 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:56.852578 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:56.853375 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:56.856064 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:56.856539 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:56.856700 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:56.860371 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:56.860381 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:56.860389 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:56.860439 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:56.864402 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:56.865430 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:56.865437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:56.865442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:56.865611 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:56.866090 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:56.866097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:56.866103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:56.866563 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:56.866747 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:56.867585 (+ 838us) server_negotiation.cc:299] Negotiation successful
0926 02:19:56.867856 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":399,"thread_start_us":320,"threads_started":1}
I20260926 02:19:56.869051 15937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:56.846248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33117 (local address 127.12.244.193:39521)
0926 02:19:56.847050 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:56.847162 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:56.849773 (+ 2611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:56.850575 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:56.850584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:56.851412 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:56.852342 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:56.852373 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:56.853557 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:56.853565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:56.855878 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:56.855898 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:56.856773 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:56.856778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:56.856996 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:56.857774 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:56.857799 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:56.860083 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:56.864639 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:56.864648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:56.864665 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:56.865261 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:56.865741 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:56.865745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:56.865747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:56.865955 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:56.866704 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:56.866711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:56.867084 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 02:19:56.867407 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 02:19:56.867839 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:19:56.870803 15934 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33117
I20260926 02:19:56.871201 15934 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:19:56.872211 15934 heartbeater.cc:511] Master 127.12.244.254:33117 requested a full tablet report, sending...
I20260926 02:19:56.874976 15745 ts_manager.cc:194] Registered new tserver with Master: 095a5d1f816640b59ea06c5b05a672d3 (127.12.244.193:41925)
I20260926 02:19:56.877234 15745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:39521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:19:56.899269 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33117
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:57.236584 15940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:57.236901 15940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:57.237056 15940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:57.244643 15940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:57.244776 15940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:57.244975 15940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:19:57.254949 15940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:57.256742 15940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:57.258330 15940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:57.273779 15945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:57.273885 15946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:57.273744 15948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:57.274847 15940 server_base.cc:1034] running on GCE node
I20260926 02:19:57.275764 15940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:57.276939 15940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:57.278211 15940 hybrid_clock.cc:648] HybridClock initialized: now 1790389197278155 us; error 52 us; skew 500 ppm
I20260926 02:19:57.278591 15940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:57 dist-test-slave-qfmw krb5kdc[15700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:57.282287 15940 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:19:57.284251 15940 webserver.cc:460] Webserver started at http://127.12.244.194:45515/ using document root <none> and password file <none>
I20260926 02:19:57.285130 15940 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:57.285293 15940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:57.285645 15940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:57.289171 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "00825104abe04259a872facbdcae6722"
format_stamp: "Formatted at 2026-09-26 02:19:57 on dist-test-slave-qfmw"
server_key: "c8ec04e54e9a39849a0b18ff1b25dd4b"
server_key_iv: "85cc91fc1465da2cf9d846fb31380b3a"
server_key_version: "encryptionkey@0"
I20260926 02:19:57.290112 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "00825104abe04259a872facbdcae6722"
format_stamp: "Formatted at 2026-09-26 02:19:57 on dist-test-slave-qfmw"
server_key: "c8ec04e54e9a39849a0b18ff1b25dd4b"
server_key_iv: "85cc91fc1465da2cf9d846fb31380b3a"
server_key_version: "encryptionkey@0"
I20260926 02:19:57.296258 15940 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:19:57.300500 15955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:57.302109 15940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:19:57.302340 15940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "00825104abe04259a872facbdcae6722"
format_stamp: "Formatted at 2026-09-26 02:19:57 on dist-test-slave-qfmw"
server_key: "c8ec04e54e9a39849a0b18ff1b25dd4b"
server_key_iv: "85cc91fc1465da2cf9d846fb31380b3a"
server_key_version: "encryptionkey@0"
I20260926 02:19:57.302568 15940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:57.321022 15940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:57.325799 15940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:57.326162 15940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:57.327337 15940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:57.328780 15940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:57.328894 15940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:57.329051 15940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:57.329123 15940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:57.364178 15940 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:42851
I20260926 02:19:57.364235 16067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:42851 every 8 connection(s)
I20260926 02:19:57.365607 15940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:19:57.370296 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 15940
I20260926 02:19:57.370503 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:19:57.371074 13267 external_mini_cluster.cc:1428] Setting key e2c62ecf64b013aeb02132d5310ff761
Sep 26 02:19:57 dist-test-slave-qfmw krb5kdc[15700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:57.391685 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:57.368466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:39611 (local address 127.12.244.254:33117)
0926 02:19:57.368655 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:57.368659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:57.370036 (+ 1377us) server_negotiation.cc:407] Connection header received
0926 02:19:57.372648 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:57.372653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:57.372730 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:57.372918 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:57.375316 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.376133 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.378896 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.379532 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.379653 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:57.383833 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:57.383844 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:57.383851 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:57.383903 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:57.387745 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:57.388688 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:57.388697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:57.388704 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:57.388891 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:57.389523 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:57.389531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:57.389535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:57.389958 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:57.390124 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:57.390925 (+ 801us) server_negotiation.cc:299] Negotiation successful
0926 02:19:57.391257 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
I20260926 02:19:57.392625 16070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:57.368921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33117 (local address 127.12.244.194:39611)
0926 02:19:57.369812 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:57.369866 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:57.372305 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:57.373240 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:57.373251 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:57.374092 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:57.375048 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:57.375082 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.376338 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.376346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:57.378666 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:57.378687 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.379816 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.379825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:57.380204 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:57.381031 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:57.381056 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:57.383531 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:57.387987 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:57.387996 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:57.388013 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:57.388514 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:57.389068 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:57.389075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:57.389081 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:57.389370 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:57.390118 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:57.390125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:57.390505 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 02:19:57.390753 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 02:19:57.391279 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":311,"threads_started":1}
I20260926 02:19:57.394359 16068 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33117
I20260926 02:19:57.394739 16068 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
I20260926 02:19:57.395861 16068 heartbeater.cc:511] Master 127.12.244.254:33117 requested a full tablet report, sending...
I20260926 02:19:57.397600 15745 ts_manager.cc:194] Registered new tserver with Master: 00825104abe04259a872facbdcae6722 (127.12.244.194:42851)
I20260926 02:19:57.398483 15745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:39611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:19:57.419541 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33117
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:37771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:57.757761 16073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:57.758122 16073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:57.758306 16073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:57.766036 16073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:57.766208 16073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:57.766390 16073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:19:57.776923 16073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:37771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:57.778293 16073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:57.779691 16073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:57.793624 16078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:57.793751 16081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:57.793780 16079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:57.794450 16073 server_base.cc:1034] running on GCE node
I20260926 02:19:57.795192 16073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:57.796226 16073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:57.797494 16073 hybrid_clock.cc:648] HybridClock initialized: now 1790389197797428 us; error 75 us; skew 500 ppm
I20260926 02:19:57.797840 16073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:57 dist-test-slave-qfmw krb5kdc[15700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:57.801409 16073 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:19:57.803061 16073 webserver.cc:460] Webserver started at http://127.12.244.195:44249/ using document root <none> and password file <none>
I20260926 02:19:57.803881 16073 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:57.804037 16073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:57.804361 16073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:57.807637 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "eeaccc3ef20440e3a1200a840d96cc6a"
format_stamp: "Formatted at 2026-09-26 02:19:57 on dist-test-slave-qfmw"
server_key: "80c33cc326b66551bd659994198a4779"
server_key_iv: "fcfe2fe90facef86bafae16b87bfc11d"
server_key_version: "encryptionkey@0"
I20260926 02:19:57.808503 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "eeaccc3ef20440e3a1200a840d96cc6a"
format_stamp: "Formatted at 2026-09-26 02:19:57 on dist-test-slave-qfmw"
server_key: "80c33cc326b66551bd659994198a4779"
server_key_iv: "fcfe2fe90facef86bafae16b87bfc11d"
server_key_version: "encryptionkey@0"
I20260926 02:19:57.814210 16073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:19:57.818288 16088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:57.819973 16073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:19:57.820183 16073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "eeaccc3ef20440e3a1200a840d96cc6a"
format_stamp: "Formatted at 2026-09-26 02:19:57 on dist-test-slave-qfmw"
server_key: "80c33cc326b66551bd659994198a4779"
server_key_iv: "fcfe2fe90facef86bafae16b87bfc11d"
server_key_version: "encryptionkey@0"
I20260926 02:19:57.820408 16073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:57.842125 16073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:57.847024 16073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:57.847374 16073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:57.848481 16073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:19:57.850107 16073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:19:57.850236 16073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:57.850376 16073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:19:57.850435 16073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:57.880630 15934 heartbeater.cc:503] Master 127.12.244.254:33117 was elected leader, sending a full tablet report...
I20260926 02:19:57.886144 16073 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:41139
I20260926 02:19:57.886199 16200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:41139 every 8 connection(s)
I20260926 02:19:57.887598 16073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:19:57.889601 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16073
I20260926 02:19:57.889863 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:19:57.890547 13267 external_mini_cluster.cc:1428] Setting key aae916e90c9c4f7b974fb3be33a06d53
Sep 26 02:19:57 dist-test-slave-qfmw krb5kdc[15700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:57.910168 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:57.890482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:33217 (local address 127.12.244.254:33117)
0926 02:19:57.890699 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:57.890705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:57.892339 (+ 1634us) server_negotiation.cc:407] Connection header received
0926 02:19:57.894772 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:57.894776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:57.894896 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:57.895024 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:57.896677 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.897434 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.899212 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.899617 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.899753 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:57.903596 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:57.903602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:57.903606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:57.903639 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:57.907174 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:57.907938 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:57.907950 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:57.907952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:57.908083 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:57.908531 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:57.908534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:57.908536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:57.908830 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:57.908939 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:57.909699 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 02:19:57.909877 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:19:57.911288 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:57.891026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33117 (local address 127.12.244.195:33217)
0926 02:19:57.892176 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 02:19:57.892227 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:19:57.894493 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:19:57.895311 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:19:57.895321 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:19:57.895878 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:19:57.896499 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:57.896517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.897608 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.897612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:57.898911 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:19:57.898924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:19:57.899873 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:19:57.899880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:19:57.900135 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:57.900799 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:19:57.900826 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:19:57.903330 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:19:57.907313 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:57.907321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:57.907337 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:57.907811 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:57.908196 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:19:57.908199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:19:57.908202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:19:57.908392 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:19:57.908933 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:19:57.908941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:19:57.909299 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 02:19:57.909561 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 02:19:57.909989 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":379,"threads_started":1}
I20260926 02:19:57.912832 16201 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33117
I20260926 02:19:57.913178 16201 heartbeater.cc:464] Registering TS with master...
I20260926 02:19:57.913975 16201 heartbeater.cc:511] Master 127.12.244.254:33117 requested a full tablet report, sending...
I20260926 02:19:57.915683 15745 ts_manager.cc:194] Registered new tserver with Master: eeaccc3ef20440e3a1200a840d96cc6a (127.12.244.195:41139)
I20260926 02:19:57.916530 15745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:33217
I20260926 02:19:57.919646 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:19:57.953272 15936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:57.950163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35580 (local address 127.12.244.254:33117)
0926 02:19:57.950431 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:57.950437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:57.950830 (+ 393us) server_negotiation.cc:407] Connection header received
0926 02:19:57.951240 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:57.951245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:57.951333 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:57.951430 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:57.952989 (+ 1559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35580: BlockingRecv error: recv got EOF from 127.0.0.1:35580 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:19:57.956805 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15806
W20260926 02:19:57.960158 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15806: Not authorized: Client connection negotiation failed: client connection to 127.12.244.193:41925: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:19:57.960367 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 15806
W20260926 02:19:57.960441 16212 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:19:57.957898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.12.244.193:41925)
0926 02:19:57.958552 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:57.958562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:57.958598 (+ 36us) server_negotiation.cc:407] Connection header received
0926 02:19:57.958682 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:57.958696 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:57.958950 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:57.959163 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:57.960065 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45236: BlockingRecv error: recv got EOF from 127.0.0.1:45236 (error 108)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":301,"threads_started":1}
I20260926 02:19:57.976680 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15940
W20260926 02:19:57.979526 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15940: Not authorized: Client connection negotiation failed: client connection to 127.12.244.194:42851: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:19:57.979717 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 15940
I20260926 02:19:57.995211 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16073
W20260926 02:19:57.998406 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16073: Not authorized: Client connection negotiation failed: client connection to 127.12.244.195:41139: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:19:57.998574 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16073
I20260926 02:19:58.013978 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 15713
I20260926 02:19:58.015394 15775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:19:58.119168 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 15713
2026-09-26T02:19:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5974 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:19:58 dist-test-slave-qfmw krb5kdc[16218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:19:58 dist-test-slave-qfmw krb5kdc[16218](info): set up 2 sockets
Sep 26 02:19:58 dist-test-slave-qfmw krb5kdc[16218](info): commencing operation
krb5kdc: starting...
W20260926 02:19:59.230588 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:19:59 dist-test-slave-qfmw krb5kdc[16218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:19:59.442626 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39585
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:19:59.784809 16231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:19:59.785179 16231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:19:59.785356 16231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:19:59.792865 16231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:19:59.793035 16231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:19:59.793154 16231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:19:59.793247 16231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:19:59.793336 16231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:19:59.803669 16231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:19:59.805088 16231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:19:59.806491 16231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:19:59.818981 16236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:59.819037 16239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:19:59.819058 16237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:19:59.820073 16231 server_base.cc:1034] running on GCE node
I20260926 02:19:59.820909 16231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:19:59.822499 16231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:19:59.823740 16231 hybrid_clock.cc:648] HybridClock initialized: now 1790389199823699 us; error 44 us; skew 500 ppm
I20260926 02:19:59.824180 16231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:19:59 dist-test-slave-qfmw krb5kdc[16218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389199, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:19:59.827576 16231 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:19:59.829185 16231 webserver.cc:460] Webserver started at http://127.12.244.254:41719/ using document root <none> and password file <none>
I20260926 02:19:59.829907 16231 fs_manager.cc:359] Metadata directory not provided
I20260926 02:19:59.830030 16231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:19:59.830291 16231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:19:59.833563 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "6810ac4b495f48be8cfce4aa76965345"
format_stamp: "Formatted at 2026-09-26 02:19:59 on dist-test-slave-qfmw"
server_key: "46859bda5479c281dde6b0968aa5245d"
server_key_iv: "2205529ee6b496d2b659518c6ca52f62"
server_key_version: "encryptionkey@0"
I20260926 02:19:59.834388 16231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "6810ac4b495f48be8cfce4aa76965345"
format_stamp: "Formatted at 2026-09-26 02:19:59 on dist-test-slave-qfmw"
server_key: "46859bda5479c281dde6b0968aa5245d"
server_key_iv: "2205529ee6b496d2b659518c6ca52f62"
server_key_version: "encryptionkey@0"
I20260926 02:19:59.839887 16231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:19:59.843677 16246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:19:59.845355 16231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:19:59.845582 16231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "6810ac4b495f48be8cfce4aa76965345"
format_stamp: "Formatted at 2026-09-26 02:19:59 on dist-test-slave-qfmw"
server_key: "46859bda5479c281dde6b0968aa5245d"
server_key_iv: "2205529ee6b496d2b659518c6ca52f62"
server_key_version: "encryptionkey@0"
I20260926 02:19:59.845805 16231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:19:59.871793 16231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:19:59.876385 16231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:19:59.876717 16231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:19:59.895892 16231 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:39585
I20260926 02:19:59.895911 16297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:39585 every 8 connection(s)
I20260926 02:19:59.897444 16231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:19:59.901481 16298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:19:59.903496 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16231
I20260926 02:19:59.903713 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:19:59.904506 13267 external_mini_cluster.cc:1428] Setting key 6cafb1f07e53e8abf7cc9abca08f0e77
I20260926 02:19:59.916307 16298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345: Bootstrap starting.
Sep 26 02:19:59 dist-test-slave-qfmw krb5kdc[16218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:19:59.919999 16298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345: Neither blocks nor log segments found. Creating new log.
I20260926 02:19:59.921450 16298 log.cc:824] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345: Log is configured to *not* fsync() on all Append() calls
I20260926 02:19:59.924850 16298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345: No bootstrap required, opened a new log
I20260926 02:19:59.927805 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:19:59.906641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.12.244.254:39585)
0926 02:19:59.907569 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0926 02:19:59.907582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:19:59.907623 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:19:59.909253 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:19:59.909280 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:19:59.909756 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:19:59.910184 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:19:59.911834 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:59.913243 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:59.914549 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:19:59.915412 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:19:59.915622 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:19:59.919884 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:19:59.919897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:19:59.919916 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:19:59.919960 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:19:59.923398 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:59.924175 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:59.924182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:59.924190 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:59.924353 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:19:59.924733 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:19:59.924738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:19:59.924742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:19:59.925373 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:19:59.925712 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 02:19:59.926149 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 02:19:59.926606 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":434,"threads_started":1}
I20260926 02:19:59.932251 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:19:59.934253 16298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6810ac4b495f48be8cfce4aa76965345" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39585 } }
I20260926 02:19:59.934725 16298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:19:59.934854 16298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810ac4b495f48be8cfce4aa76965345, State: Initialized, Role: FOLLOWER
I20260926 02:19:59.935590 16298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6810ac4b495f48be8cfce4aa76965345" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39585 } }
I20260926 02:19:59.935864 16298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:19:59.936030 16298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:19:59.936232 16298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:19:59.938678 16298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6810ac4b495f48be8cfce4aa76965345" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39585 } }
I20260926 02:19:59.939375 16298 leader_election.cc:302] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6810ac4b495f48be8cfce4aa76965345; no voters:
I20260926 02:19:59.940271 16298 leader_election.cc:288] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:19:59.940403 16304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:19:59.941946 16304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [term 1 LEADER]: Becoming Leader. State: Replica: 6810ac4b495f48be8cfce4aa76965345, State: Running, Role: LEADER
I20260926 02:19:59.942629 16304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6810ac4b495f48be8cfce4aa76965345" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39585 } }
I20260926 02:19:59.943078 16298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:19:59.945700 16306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6810ac4b495f48be8cfce4aa76965345. Latest consensus state: current_term: 1 leader_uuid: "6810ac4b495f48be8cfce4aa76965345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6810ac4b495f48be8cfce4aa76965345" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39585 } } }
I20260926 02:19:59.946112 16306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:59.946692 16305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6810ac4b495f48be8cfce4aa76965345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6810ac4b495f48be8cfce4aa76965345" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39585 } } }
I20260926 02:19:59.947038 16305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345 [sys.catalog]: This master's current role is: LEADER
I20260926 02:19:59.949409 16309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:19:59.958711 16309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:19:59.971199 16309 catalog_manager.cc:1347] Generated new cluster ID: 055216b499fb46809d92804ad429dcbf
I20260926 02:19:59.971338 16309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:00.003890 16309 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:00.006016 16309 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:00.022502 16329 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:20:00.029147 16309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6810ac4b495f48be8cfce4aa76965345: Generated new TSK 0
I20260926 02:20:00.030041 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:00.011736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.12.244.254:39585)
0926 02:20:00.012010 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:00.012020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:00.021068 (+ 9048us) server_negotiation.cc:407] Connection header received
0926 02:20:00.021729 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:00.021739 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:00.021859 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:00.021957 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:00.023920 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:00.024685 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:00.026106 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:00.026572 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:00.026722 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:00.028027 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:00.028032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:00.028036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:00.028909 (+ 873us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:00.028964 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:00.028988 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:00.029137 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:00.029543 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 02:20:00.029667 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:20:00.030433 16309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:20:00.041071 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16231
I20260926 02:20:00.042500 16293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:00.136462 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16231
2026-09-26T02:20:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2012 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:00 dist-test-slave-qfmw krb5kdc[16333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:00 dist-test-slave-qfmw krb5kdc[16333](info): set up 2 sockets
Sep 26 02:20:00 dist-test-slave-qfmw krb5kdc[16333](info): commencing operation
krb5kdc: starting...
W20260926 02:20:01.234587 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:01 dist-test-slave-qfmw krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:20:01.445091 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33537
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:33537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:01.786119 16346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:01.786513 16346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:01.786690 16346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:01.794179 16346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:01.794361 16346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:01.794476 16346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:01.794584 16346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:01.794673 16346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:01.804976 16346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:33537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:01.806423 16346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:01.807837 16346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:01.820204 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:01.820248 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:01.820268 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:01.821343 16346 server_base.cc:1034] running on GCE node
I20260926 02:20:01.822201 16346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:01.823864 16346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:01.825125 16346 hybrid_clock.cc:648] HybridClock initialized: now 1790389201825079 us; error 68 us; skew 500 ppm
I20260926 02:20:01.825507 16346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:01 dist-test-slave-qfmw krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389201, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:01.835597 16346 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:20:01.837435 16346 webserver.cc:460] Webserver started at http://127.12.244.254:36981/ using document root <none> and password file <none>
I20260926 02:20:01.838199 16346 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:01.838366 16346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:01.838706 16346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:01.841919 16346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "01eeb25c016e418eaa43221e09d0afc0"
format_stamp: "Formatted at 2026-09-26 02:20:01 on dist-test-slave-qfmw"
server_key: "2d39deecae6f03183d6c1964e954753d"
server_key_iv: "b1cd6c4e1521cbefdc56952b93e2abff"
server_key_version: "encryptionkey@0"
I20260926 02:20:01.842756 16346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "01eeb25c016e418eaa43221e09d0afc0"
format_stamp: "Formatted at 2026-09-26 02:20:01 on dist-test-slave-qfmw"
server_key: "2d39deecae6f03183d6c1964e954753d"
server_key_iv: "b1cd6c4e1521cbefdc56952b93e2abff"
server_key_version: "encryptionkey@0"
I20260926 02:20:01.848450 16346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:20:01.852546 16361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:01.854116 16346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:01.854354 16346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "01eeb25c016e418eaa43221e09d0afc0"
format_stamp: "Formatted at 2026-09-26 02:20:01 on dist-test-slave-qfmw"
server_key: "2d39deecae6f03183d6c1964e954753d"
server_key_iv: "b1cd6c4e1521cbefdc56952b93e2abff"
server_key_version: "encryptionkey@0"
I20260926 02:20:01.854624 16346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:01.879963 16346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:01.884407 16346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:01.884752 16346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:01.904660 16346 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:33537
I20260926 02:20:01.904681 16412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:33537 every 8 connection(s)
I20260926 02:20:01.906179 16346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:01.910190 16413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:01.915378 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16346
I20260926 02:20:01.915602 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:01.916399 13267 external_mini_cluster.cc:1428] Setting key 0713f4c6844529321746334ec37e5f17
I20260926 02:20:01.922956 16413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0: Bootstrap starting.
I20260926 02:20:01.926254 16413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:01.927430 16413 log.cc:824] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:01.931341 16413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0: No bootstrap required, opened a new log
Sep 26 02:20:01 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:01.942481 16413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eeb25c016e418eaa43221e09d0afc0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33537 } }
I20260926 02:20:01.942971 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:01.918773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.12.244.254:33537)
0926 02:20:01.920070 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:01.920083 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:01.920130 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:20:01.921897 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:01.921923 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:01.922386 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:01.922815 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:01.924273 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:01.925613 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:01.926929 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:01.927807 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:01.928041 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:01.932961 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:01.932974 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:01.932996 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:01.933059 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:01.937516 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:01.938411 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:01.938422 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:01.938433 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:01.938688 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:01.939298 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:01.939303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:01.939307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:01.940146 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:01.940538 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:01.940928 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 02:20:01.941452 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1019,"thread_start_us":752,"threads_started":1}
I20260926 02:20:01.943012 16413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:01.943142 16413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01eeb25c016e418eaa43221e09d0afc0, State: Initialized, Role: FOLLOWER
I20260926 02:20:01.943761 16413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eeb25c016e418eaa43221e09d0afc0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33537 } }
I20260926 02:20:01.943992 16413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:01.944090 16413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:01.944214 16413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:01.947297 16413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eeb25c016e418eaa43221e09d0afc0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33537 } }
I20260926 02:20:01.947850 16413 leader_election.cc:302] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01eeb25c016e418eaa43221e09d0afc0; no voters:
I20260926 02:20:01.948559 16413 leader_election.cc:288] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:01.948714 16418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:01.949548 16418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [term 1 LEADER]: Becoming Leader. State: Replica: 01eeb25c016e418eaa43221e09d0afc0, State: Running, Role: LEADER
I20260926 02:20:01.950229 16418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eeb25c016e418eaa43221e09d0afc0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33537 } }
I20260926 02:20:01.950963 16413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:01.953506 16419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01eeb25c016e418eaa43221e09d0afc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eeb25c016e418eaa43221e09d0afc0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33537 } } }
I20260926 02:20:01.953481 16420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01eeb25c016e418eaa43221e09d0afc0. Latest consensus state: current_term: 1 leader_uuid: "01eeb25c016e418eaa43221e09d0afc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eeb25c016e418eaa43221e09d0afc0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33537 } } }
I20260926 02:20:01.954011 16420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:01.954011 16419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:01.955431 16426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:01.961233 16426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:01.974074 16426 catalog_manager.cc:1347] Generated new cluster ID: 699f0b161ea44ac99048ac8737995d26
I20260926 02:20:01.974225 16426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:01.995384 16426 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:01.997236 16426 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:02.005357 16426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01eeb25c016e418eaa43221e09d0afc0: Generated new TSK 0
I20260926 02:20:02.006626 16426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:20:02.073421 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33537
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:02.428143 16439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:02.428495 16439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:02.428669 16439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:02.436316 16439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:02.436488 16439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:02.436681 16439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:02.446977 16439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:02.448465 16439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:02.449846 16439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:02.464277 16445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:02.464234 16444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:02.464874 16447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:02.465430 16439 server_base.cc:1034] running on GCE node
I20260926 02:20:02.466137 16439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:02.467335 16439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:02.468595 16439 hybrid_clock.cc:648] HybridClock initialized: now 1790389202468525 us; error 64 us; skew 500 ppm
I20260926 02:20:02.469003 16439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:02 dist-test-slave-qfmw krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:02.472848 16439 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:20:02.474576 16439 webserver.cc:460] Webserver started at http://127.12.244.193:35357/ using document root <none> and password file <none>
I20260926 02:20:02.475440 16439 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:02.475577 16439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:02.475943 16439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:02.479154 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "c1aed053c0dd46d7b69cc8cc896e5bf4"
format_stamp: "Formatted at 2026-09-26 02:20:02 on dist-test-slave-qfmw"
server_key: "1e58c2f6503584fe4c27ddee6364c9d5"
server_key_iv: "93d61f001d582e4170c267924d2716ca"
server_key_version: "encryptionkey@0"
I20260926 02:20:02.480077 16439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "c1aed053c0dd46d7b69cc8cc896e5bf4"
format_stamp: "Formatted at 2026-09-26 02:20:02 on dist-test-slave-qfmw"
server_key: "1e58c2f6503584fe4c27ddee6364c9d5"
server_key_iv: "93d61f001d582e4170c267924d2716ca"
server_key_version: "encryptionkey@0"
I20260926 02:20:02.485862 16439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:20:02.489849 16454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:02.491665 16439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:20:02.491876 16439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "c1aed053c0dd46d7b69cc8cc896e5bf4"
format_stamp: "Formatted at 2026-09-26 02:20:02 on dist-test-slave-qfmw"
server_key: "1e58c2f6503584fe4c27ddee6364c9d5"
server_key_iv: "93d61f001d582e4170c267924d2716ca"
server_key_version: "encryptionkey@0"
I20260926 02:20:02.492125 16439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:02.515486 16439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:02.521654 16439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:02.522030 16439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:02.523239 16439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:02.524710 16439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:02.524791 16439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:02.524924 16439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:02.524988 16439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:02.559847 16439 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:36269
I20260926 02:20:02.559888 16566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:36269 every 8 connection(s)
I20260926 02:20:02.561290 16439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:02.568053 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16439
I20260926 02:20:02.568271 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:02.568955 13267 external_mini_cluster.cc:1428] Setting key 3472e8dc7a1faed4660df7c4494ee3ff
Sep 26 02:20:02 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389202, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:02.586544 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:02.564004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:57269 (local address 127.12.244.254:33537)
0926 02:20:02.564478 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:02.564483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:02.565455 (+ 972us) server_negotiation.cc:407] Connection header received
0926 02:20:02.567626 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:02.567632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:02.567734 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:02.567860 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:02.569756 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:02.570572 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:02.573365 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:02.573783 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:02.573983 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:02.578951 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:02.578962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:02.578971 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:02.579021 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:02.582790 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:02.583816 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:02.583825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:02.583831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:02.584006 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:02.584468 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:02.584476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:02.584483 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:02.584875 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:02.585043 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:02.585786 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 02:20:02.586077 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":245,"threads_started":1}
I20260926 02:20:02.587546 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:02.564489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33537 (local address 127.12.244.193:57269)
0926 02:20:02.565264 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:02.565309 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:02.567279 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:02.568163 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:02.568175 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:02.568789 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:02.569506 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:02.569537 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:02.570769 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:02.570783 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:02.573139 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:02.573160 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:02.574126 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:02.574135 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:02.574507 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:02.575428 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:02.575472 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:02.578650 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:02.583014 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:02.583024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:02.583042 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:02.583648 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:02.584133 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:02.584137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:02.584140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:02.584341 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:02.585004 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:02.585011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:02.585368 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 02:20:02.585618 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:20:02.586207 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":278,"threads_started":1}
I20260926 02:20:02.589370 16567 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33537
I20260926 02:20:02.589722 16567 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:20:02.590588 16567 heartbeater.cc:511] Master 127.12.244.254:33537 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:20:02.593179 16378 ts_manager.cc:194] Registered new tserver with Master: c1aed053c0dd46d7b69cc8cc896e5bf4 (127.12.244.193:36269)
I20260926 02:20:02.594750 16378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:57269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:20:02.618805 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33537
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:02.972373 16573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:02.972718 16573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:02.972892 16573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:02.981079 16573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:02.981231 16573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:02.981442 16573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:02.992278 16573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:02.993707 16573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:02.995177 16573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:03.010113 16578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:03.010267 16579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:03.010263 16581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:03.011902 16573 server_base.cc:1034] running on GCE node
I20260926 02:20:03.012646 16573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:03.013698 16573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:03.014950 16573 hybrid_clock.cc:648] HybridClock initialized: now 1790389203014908 us; error 36 us; skew 500 ppm
I20260926 02:20:03.015349 16573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:03.018882 16573 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:20:03.020711 16573 webserver.cc:460] Webserver started at http://127.12.244.194:41651/ using document root <none> and password file <none>
I20260926 02:20:03.021467 16573 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:03.021617 16573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:03.021943 16573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:03.025092 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "a02245818da74ca896ed75f808d7fce6"
format_stamp: "Formatted at 2026-09-26 02:20:03 on dist-test-slave-qfmw"
server_key: "d477dfa53383732ebc93937f6a1c21d1"
server_key_iv: "782769dc701d40db04fa41cfc3d94624"
server_key_version: "encryptionkey@0"
I20260926 02:20:03.025954 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "a02245818da74ca896ed75f808d7fce6"
format_stamp: "Formatted at 2026-09-26 02:20:03 on dist-test-slave-qfmw"
server_key: "d477dfa53383732ebc93937f6a1c21d1"
server_key_iv: "782769dc701d40db04fa41cfc3d94624"
server_key_version: "encryptionkey@0"
I20260926 02:20:03.031761 16573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:20:03.035706 16588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:03.037364 16573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:20:03.037609 16573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "a02245818da74ca896ed75f808d7fce6"
format_stamp: "Formatted at 2026-09-26 02:20:03 on dist-test-slave-qfmw"
server_key: "d477dfa53383732ebc93937f6a1c21d1"
server_key_iv: "782769dc701d40db04fa41cfc3d94624"
server_key_version: "encryptionkey@0"
I20260926 02:20:03.037849 16573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:03.063841 16573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:03.069499 16573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:03.069862 16573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:03.071008 16573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:03.072908 16573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:03.073027 16573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:03.073184 16573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:03.073257 16573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:03.109334 16573 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:45607
I20260926 02:20:03.109429 16700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:45607 every 8 connection(s)
I20260926 02:20:03.110777 16573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:03.120142 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16573
I20260926 02:20:03.120332 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:03.120795 13267 external_mini_cluster.cc:1428] Setting key fe5df58f19a9590496b9b95540360bfb
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:03.133865 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.113548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:58371 (local address 127.12.244.254:33537)
0926 02:20:03.114042 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:03.114047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:03.114928 (+ 881us) server_negotiation.cc:407] Connection header received
0926 02:20:03.116909 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:03.116915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:03.117017 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:03.117138 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:03.118771 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.119621 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.121563 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.122134 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.122279 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.126490 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:03.126501 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:03.126509 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:03.126562 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:03.129836 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.130986 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.130992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.130994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.131254 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.131893 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.131897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.131899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.132183 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:03.132308 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:03.133357 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0926 02:20:03.133554 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":365,"thread_start_us":280,"threads_started":1}
I20260926 02:20:03.135551 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.114032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33537 (local address 127.12.244.194:58371)
0926 02:20:03.114756 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:03.114796 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:03.116583 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:03.117406 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:03.117416 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:03.117970 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:03.118571 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:03.118588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.119787 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.119792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:03.121296 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:03.121313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.122426 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.122434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:03.122799 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.123757 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:03.123784 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:03.126170 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:03.130036 (+ 3866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:03.130050 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:03.130074 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:03.130809 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:03.131388 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:03.131396 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:03.131402 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:03.131689 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:03.132349 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:03.132362 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:03.132900 (+ 538us) client_negotiation.cc:769] Sending connection context
0926 02:20:03.133248 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 02:20:03.133927 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":280,"threads_started":1}
I20260926 02:20:03.137344 16701 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33537
I20260926 02:20:03.137729 16701 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:03.138823 16701 heartbeater.cc:511] Master 127.12.244.254:33537 requested a full tablet report, sending...
I20260926 02:20:03.140846 16378 ts_manager.cc:194] Registered new tserver with Master: a02245818da74ca896ed75f808d7fce6 (127.12.244.194:45607)
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
I20260926 02:20:03.142164 16378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:58371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:20:03.172293 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33537
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:03.526010 16707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:03.526351 16707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:03.526510 16707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:03.534405 16707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:03.534590 16707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:03.534801 16707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:03.545162 16707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:03.546674 16707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:03.548372 16707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:03.562870 16712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:03.562876 16713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:03.564262 16707 server_base.cc:1034] running on GCE node
W20260926 02:20:03.566233 16715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:03.567070 16707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:03.568186 16707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:03.569423 16707 hybrid_clock.cc:648] HybridClock initialized: now 1790389203569375 us; error 38 us; skew 500 ppm
I20260926 02:20:03.569727 16707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:03.573359 16707 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:20:03.575191 16707 webserver.cc:460] Webserver started at http://127.12.244.195:40507/ using document root <none> and password file <none>
I20260926 02:20:03.575919 16707 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:03.576038 16707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:03.576310 16707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:03.579454 16707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "888308fd91384a4f84e26bc0f828c9eb"
format_stamp: "Formatted at 2026-09-26 02:20:03 on dist-test-slave-qfmw"
server_key: "f4d75cfcc12af9a46ccd6acc0d7443c7"
server_key_iv: "817436dbdf602830dd9120c57ca2e4e1"
server_key_version: "encryptionkey@0"
I20260926 02:20:03.580236 16707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "888308fd91384a4f84e26bc0f828c9eb"
format_stamp: "Formatted at 2026-09-26 02:20:03 on dist-test-slave-qfmw"
server_key: "f4d75cfcc12af9a46ccd6acc0d7443c7"
server_key_iv: "817436dbdf602830dd9120c57ca2e4e1"
server_key_version: "encryptionkey@0"
I20260926 02:20:03.585788 16707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:20:03.589772 16722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:03.591495 16707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:20:03.591688 16707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "888308fd91384a4f84e26bc0f828c9eb"
format_stamp: "Formatted at 2026-09-26 02:20:03 on dist-test-slave-qfmw"
server_key: "f4d75cfcc12af9a46ccd6acc0d7443c7"
server_key_iv: "817436dbdf602830dd9120c57ca2e4e1"
server_key_version: "encryptionkey@0"
I20260926 02:20:03.591893 16707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:03.598917 16567 heartbeater.cc:503] Master 127.12.244.254:33537 was elected leader, sending a full tablet report...
I20260926 02:20:03.609124 16707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:03.614030 16707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:03.614418 16707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:03.615522 16707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:03.616987 16707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:03.617096 16707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:03.617254 16707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:03.617326 16707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:03.652411 16707 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:45391
I20260926 02:20:03.652503 16834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:45391 every 8 connection(s)
I20260926 02:20:03.653924 16707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:03.655498 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16707
I20260926 02:20:03.655675 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:03.656229 13267 external_mini_cluster.cc:1428] Setting key defd76d6eb00d38e46e740e6275e69ed
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:03.676852 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.656731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:54817 (local address 127.12.244.254:33537)
0926 02:20:03.657281 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:03.657288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:03.660006 (+ 2718us) server_negotiation.cc:407] Connection header received
0926 02:20:03.661900 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:03.661905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:03.661988 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:03.662108 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:03.663851 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.664684 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.666380 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.666767 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.666873 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.670670 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:03.670675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:03.670679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:03.670714 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:03.673654 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.674475 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.674480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.674482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.674614 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.675176 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.675181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.675183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.675503 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:03.675633 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:03.676396 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 02:20:03.676583 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":246,"threads_started":1}
I20260926 02:20:03.677830 16838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.657487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33537 (local address 127.12.244.195:54817)
0926 02:20:03.659810 (+ 2323us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:03.659891 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:03.661608 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:03.662343 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:03.662352 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:03.662888 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:03.663648 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:03.663667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.664833 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.664837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:03.666222 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:03.666240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.666992 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.666997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:03.667269 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.668051 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:03.668074 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:03.670451 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:03.673824 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:03.673832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:03.673849 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:03.674342 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:03.674712 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:03.674715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:03.674717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:03.674967 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:03.675620 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:03.675627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:03.676012 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 02:20:03.676246 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 02:20:03.676690 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"thread_start_us":1810,"threads_started":1}
I20260926 02:20:03.679435 16835 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33537
I20260926 02:20:03.679854 16835 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:03.680692 16835 heartbeater.cc:511] Master 127.12.244.254:33537 requested a full tablet report, sending...
I20260926 02:20:03.682268 16378 ts_manager.cc:194] Registered new tserver with Master: 888308fd91384a4f84e26bc0f828c9eb (127.12.244.195:45391)
I20260926 02:20:03.683363 16378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:54817
I20260926 02:20:03.684876 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:03.735948 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.721153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.12.244.254:33537)
0926 02:20:03.721370 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:03.721374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:03.721872 (+ 498us) server_negotiation.cc:407] Connection header received
0926 02:20:03.722068 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:03.722072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:03.722144 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:03.722245 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:03.723817 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.724578 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.725841 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.726274 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.726378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.730355 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:03.730360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:03.730364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:03.730404 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:03.733161 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.733910 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.733915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.733917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.734047 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.734488 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.734493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.734496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.734873 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:03.735088 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:03.735511 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 02:20:03.735693 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260926 02:20:03.736581 16378 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54208
I20260926 02:20:03.738962 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16439
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:20:03.756731 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.739971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.12.244.193:36269)
0926 02:20:03.740582 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:03.740590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:03.740615 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:20:03.740670 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:03.740676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:03.740891 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:03.741021 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:03.742403 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.743531 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.744825 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.745455 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.745570 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.749847 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:03.749864 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:03.749881 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:03.749935 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:03.753421 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.754331 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.754340 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.754343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.754490 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.754935 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.754939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.754942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.755435 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:03.755671 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:03.756031 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 02:20:03.756350 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":289,"threads_started":1}
W20260926 02:20:03.758103 16562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57900
W20260926 02:20:03.759467 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:20:03.759665 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16439
I20260926 02:20:03.775856 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16573
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:20:03.793133 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.776954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41288 (local address 127.12.244.194:45607)
0926 02:20:03.777545 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:03.777552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:03.777583 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:20:03.777642 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:03.777649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:03.777913 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:03.778050 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:03.779382 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.780415 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.781706 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.782306 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.782420 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.786631 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:03.786648 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:03.786666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:03.786707 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:03.789888 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.790682 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.790691 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.790694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.790883 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.791436 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.791440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.791442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.791829 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:03.792158 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:03.792456 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 02:20:03.792694 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":284,"threads_started":1}
W20260926 02:20:03.794482 16696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41288
W20260926 02:20:03.795773 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16573: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:20:03.795995 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16573
I20260926 02:20:03.811671 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16707
Sep 26 02:20:03 dist-test-slave-qfmw krb5kdc[16333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389203, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:20:03.828804 16849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:03.812625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.12.244.195:45391)
0926 02:20:03.813253 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:03.813260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:03.813291 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:20:03.813350 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:03.813357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:03.813589 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:03.813783 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:03.815213 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.816277 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.817597 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:03.818185 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:03.818332 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:03.822865 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:03.822880 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:03.822898 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:03.822940 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:03.825736 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.826551 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.826560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.826562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.826692 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:03.827125 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:03.827131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:03.827134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:03.827509 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:03.827829 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:03.828143 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 02:20:03.828436 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":345,"threads_started":1}
W20260926 02:20:03.830186 16830 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50748
W20260926 02:20:03.831614 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16707: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:20:03.831807 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16707
I20260926 02:20:03.847332 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16346
I20260926 02:20:03.848588 16408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:03.953320 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16346
2026-09-26T02:20:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3823 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:04 dist-test-slave-qfmw krb5kdc[16853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:04 dist-test-slave-qfmw krb5kdc[16853](info): set up 2 sockets
Sep 26 02:20:04 dist-test-slave-qfmw krb5kdc[16853](info): commencing operation
krb5kdc: starting...
W20260926 02:20:05.074208 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:05 dist-test-slave-qfmw krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:20:05.281250 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34991
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:34991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:05.618731 16866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:05.619060 16866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:05.619233 16866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:05.626660 16866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:05.626829 16866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:05.626880 16866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:05.626936 16866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:05.626982 16866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:05.637885 16866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:34991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:05.639241 16866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:05.640570 16866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:05.652894 16871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:05.653024 16874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:05.653002 16872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:05.654426 16866 server_base.cc:1034] running on GCE node
I20260926 02:20:05.655375 16866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:05.656906 16866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:05.658169 16866 hybrid_clock.cc:648] HybridClock initialized: now 1790389205658110 us; error 57 us; skew 500 ppm
I20260926 02:20:05.658571 16866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:05 dist-test-slave-qfmw krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389205, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:05.662201 16866 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:20:05.664000 16866 webserver.cc:460] Webserver started at http://127.12.244.254:35391/ using document root <none> and password file <none>
I20260926 02:20:05.664796 16866 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:05.664945 16866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:05.665268 16866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:05.668512 16866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "1b44fd19d12942d5a9d86e26619ad0dd"
format_stamp: "Formatted at 2026-09-26 02:20:05 on dist-test-slave-qfmw"
server_key: "25a1b51142e5cf6a2252ebb7bce27276"
server_key_iv: "4897d08476866b4d49511c50d0db1195"
server_key_version: "encryptionkey@0"
I20260926 02:20:05.669368 16866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "1b44fd19d12942d5a9d86e26619ad0dd"
format_stamp: "Formatted at 2026-09-26 02:20:05 on dist-test-slave-qfmw"
server_key: "25a1b51142e5cf6a2252ebb7bce27276"
server_key_iv: "4897d08476866b4d49511c50d0db1195"
server_key_version: "encryptionkey@0"
I20260926 02:20:05.674873 16866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:20:05.678745 16881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:05.680359 16866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:20:05.680593 16866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "1b44fd19d12942d5a9d86e26619ad0dd"
format_stamp: "Formatted at 2026-09-26 02:20:05 on dist-test-slave-qfmw"
server_key: "25a1b51142e5cf6a2252ebb7bce27276"
server_key_iv: "4897d08476866b4d49511c50d0db1195"
server_key_version: "encryptionkey@0"
I20260926 02:20:05.680827 16866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:05.699957 16866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:05.704726 16866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:05.705061 16866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:05.725015 16866 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:34991
I20260926 02:20:05.725033 16932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:34991 every 8 connection(s)
I20260926 02:20:05.726521 16866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:05.730893 16933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:05.732002 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16866
I20260926 02:20:05.732245 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:05.732901 13267 external_mini_cluster.cc:1428] Setting key 0f8b9f3b68cfe5400878c19d96c8585c
I20260926 02:20:05.747049 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd: Bootstrap starting.
Sep 26 02:20:05 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:05.750943 16933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:05.752722 16933 log.cc:824] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:05.756767 16933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd: No bootstrap required, opened a new log
I20260926 02:20:05.759879 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:05.735255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.12.244.254:34991)
0926 02:20:05.736189 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:05.736206 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:05.736280 (+ 74us) server_negotiation.cc:407] Connection header received
0926 02:20:05.738428 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:05.738465 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:05.739134 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:05.739684 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:05.741063 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:05.743221 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:05.744583 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:05.745792 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:05.745981 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:05.750095 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:05.750108 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:05.750130 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:05.750188 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:05.754501 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:05.755400 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:05.755410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:05.755420 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:05.755666 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:05.756167 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:05.756173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:05.756179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:05.756990 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:05.757351 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:05.757756 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 02:20:05.758256 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":365,"threads_started":1}
I20260926 02:20:05.766908 16933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34991 } }
I20260926 02:20:05.767407 16933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:05.767503 16933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b44fd19d12942d5a9d86e26619ad0dd, State: Initialized, Role: FOLLOWER
I20260926 02:20:05.768143 16933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34991 } }
I20260926 02:20:05.768392 16933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:05.768504 16933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:05.768643 16933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:05.771373 16933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34991 } }
I20260926 02:20:05.771921 16933 leader_election.cc:302] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b44fd19d12942d5a9d86e26619ad0dd; no voters:
I20260926 02:20:05.772571 16933 leader_election.cc:288] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:05.772774 16938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:05.773537 16938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [term 1 LEADER]: Becoming Leader. State: Replica: 1b44fd19d12942d5a9d86e26619ad0dd, State: Running, Role: LEADER
I20260926 02:20:05.774400 16938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34991 } }
I20260926 02:20:05.775250 16933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:05.777506 16940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b44fd19d12942d5a9d86e26619ad0dd. Latest consensus state: current_term: 1 leader_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34991 } } }
I20260926 02:20:05.777464 16939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b44fd19d12942d5a9d86e26619ad0dd" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34991 } } }
I20260926 02:20:05.777968 16939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:05.778545 16940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:05.778776 16943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:05.785568 16943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:05.799402 16943 catalog_manager.cc:1347] Generated new cluster ID: e786ba1aaa0748d4b4422fa0aa7e1296
I20260926 02:20:05.799542 16943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:05.829514 16943 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:05.831080 16943 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:05.837357 16943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b44fd19d12942d5a9d86e26619ad0dd: Generated new TSK 0
I20260926 02:20:05.838503 16943 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:20:05.898581 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34991
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:06.246615 16959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:06.247045 16959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:06.247227 16959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:06.254750 16959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:06.254874 16959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:06.254997 16959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:06.265312 16959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:06.266611 16959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:06.268079 16959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:06.282151 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:06.282426 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:06.282114 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:06.283178 16959 server_base.cc:1034] running on GCE node
I20260926 02:20:06.283878 16959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:06.284946 16959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:06.286181 16959 hybrid_clock.cc:648] HybridClock initialized: now 1790389206286134 us; error 41 us; skew 500 ppm
I20260926 02:20:06.286528 16959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:06 dist-test-slave-qfmw krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:06.289860 16959 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:20:06.291590 16959 webserver.cc:460] Webserver started at http://127.12.244.193:40853/ using document root <none> and password file <none>
I20260926 02:20:06.292328 16959 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:06.292495 16959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:06.292829 16959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:06.295935 16959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "177ff1743e314df4a502660cd1987488"
format_stamp: "Formatted at 2026-09-26 02:20:06 on dist-test-slave-qfmw"
server_key: "6082079f036176da0ba7f9a97c014c4d"
server_key_iv: "b16d482d3eb1f61ab6d6904b30072fa6"
server_key_version: "encryptionkey@0"
I20260926 02:20:06.296779 16959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "177ff1743e314df4a502660cd1987488"
format_stamp: "Formatted at 2026-09-26 02:20:06 on dist-test-slave-qfmw"
server_key: "6082079f036176da0ba7f9a97c014c4d"
server_key_iv: "b16d482d3eb1f61ab6d6904b30072fa6"
server_key_version: "encryptionkey@0"
I20260926 02:20:06.302347 16959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:20:06.306406 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:06.307952 16959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 02:20:06.308145 16959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "177ff1743e314df4a502660cd1987488"
format_stamp: "Formatted at 2026-09-26 02:20:06 on dist-test-slave-qfmw"
server_key: "6082079f036176da0ba7f9a97c014c4d"
server_key_iv: "b16d482d3eb1f61ab6d6904b30072fa6"
server_key_version: "encryptionkey@0"
I20260926 02:20:06.308569 16959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:06.339701 16959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:06.343986 16959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:06.344296 16959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:06.345264 16959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:06.346625 16959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:06.346735 16959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:06.346902 16959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:06.346984 16959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:06.381536 16959 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:40949
I20260926 02:20:06.381557 17086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:40949 every 8 connection(s)
I20260926 02:20:06.382899 16959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:06.389688 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 16959
I20260926 02:20:06.389909 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:06.390558 13267 external_mini_cluster.cc:1428] Setting key 4aa82db5294b5cf0218dd383562b6667
Sep 26 02:20:06 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:06.404522 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:06.385623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:49513 (local address 127.12.244.254:34991)
0926 02:20:06.386118 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:06.386122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:06.387080 (+ 958us) server_negotiation.cc:407] Connection header received
0926 02:20:06.388949 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:06.388955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:06.389044 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:06.389172 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:06.390953 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.391771 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.393407 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.393833 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.393978 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:06.398017 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:06.398022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:06.398026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:06.398061 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:06.400895 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:06.401968 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:06.401972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:06.401975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:06.402108 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:06.402623 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:06.402626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:06.402628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:06.402948 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:06.403068 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:06.404078 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0926 02:20:06.404265 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":313,"threads_started":1}
I20260926 02:20:06.406258 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:06.386202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34991 (local address 127.12.244.193:49513)
0926 02:20:06.386898 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:06.386946 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:06.388668 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:06.389433 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:06.389442 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:06.390101 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:06.390773 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:06.390794 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.391946 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.391950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:06.393234 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:06.393246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.394080 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.394089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:06.394336 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:06.395064 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:06.395123 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:06.397687 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:06.401081 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:06.401094 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:06.401117 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:06.401824 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:06.402229 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:06.402237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:06.402243 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:06.402515 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:06.403087 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:06.403118 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:06.403648 (+ 530us) client_negotiation.cc:769] Sending connection context
0926 02:20:06.403975 (+ 327us) client_negotiation.cc:240] Negotiation successful
0926 02:20:06.404642 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":284,"threads_started":1}
I20260926 02:20:06.408160 17087 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34991
I20260926 02:20:06.408519 17087 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:06.409284 17087 heartbeater.cc:511] Master 127.12.244.254:34991 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:20:06.411677 16898 ts_manager.cc:194] Registered new tserver with Master: 177ff1743e314df4a502660cd1987488 (127.12.244.193:40949)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:20:06.413457 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:49513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:20:06.438457 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34991
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:06.779227 17093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:06.779590 17093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:06.779752 17093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:06.787708 17093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:06.787883 17093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:06.788056 17093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:06.798473 17093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:06.800093 17093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:06.801455 17093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:06.815610 17098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:06.815627 17099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:06.815610 17101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:06.816854 17093 server_base.cc:1034] running on GCE node
I20260926 02:20:06.817471 17093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:06.818459 17093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:06.819672 17093 hybrid_clock.cc:648] HybridClock initialized: now 1790389206819623 us; error 34 us; skew 500 ppm
I20260926 02:20:06.819980 17093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:06 dist-test-slave-qfmw krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:06.823246 17093 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:20:06.824864 17093 webserver.cc:460] Webserver started at http://127.12.244.194:44151/ using document root <none> and password file <none>
I20260926 02:20:06.825651 17093 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:06.825788 17093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:06.826128 17093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:06.829399 17093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "2559d44a9b604f93947846787203e8a7"
format_stamp: "Formatted at 2026-09-26 02:20:06 on dist-test-slave-qfmw"
server_key: "c1bd90ce3d29624fa654d73e44a76e4f"
server_key_iv: "5a5c733d4c0c98b1ed4cad894a5224cf"
server_key_version: "encryptionkey@0"
I20260926 02:20:06.830276 17093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "2559d44a9b604f93947846787203e8a7"
format_stamp: "Formatted at 2026-09-26 02:20:06 on dist-test-slave-qfmw"
server_key: "c1bd90ce3d29624fa654d73e44a76e4f"
server_key_iv: "5a5c733d4c0c98b1ed4cad894a5224cf"
server_key_version: "encryptionkey@0"
I20260926 02:20:06.836061 17093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:20:06.840071 17108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:06.841744 17093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:06.841996 17093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "2559d44a9b604f93947846787203e8a7"
format_stamp: "Formatted at 2026-09-26 02:20:06 on dist-test-slave-qfmw"
server_key: "c1bd90ce3d29624fa654d73e44a76e4f"
server_key_iv: "5a5c733d4c0c98b1ed4cad894a5224cf"
server_key_version: "encryptionkey@0"
I20260926 02:20:06.842245 17093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:06.872340 17093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:06.876938 17093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:06.877242 17093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:06.878294 17093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:06.879765 17093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:06.879842 17093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:06.880008 17093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:06.880066 17093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:06.914762 17093 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:46425
I20260926 02:20:06.914906 17220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:46425 every 8 connection(s)
I20260926 02:20:06.916163 17093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:06.919302 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 17093
I20260926 02:20:06.919521 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:06.920048 13267 external_mini_cluster.cc:1428] Setting key eb97bae4170348658c7efd146e8d4465
Sep 26 02:20:06 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389206, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:06.941953 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:06.918904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:56825 (local address 127.12.244.254:34991)
0926 02:20:06.919396 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:06.919403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:06.920669 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 02:20:06.922983 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:06.922988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:06.923057 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:06.923204 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:06.925585 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.926300 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.929088 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.929855 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.930022 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:06.934067 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:06.934075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:06.934082 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:06.934130 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:06.937931 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:06.938857 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:06.938865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:06.938871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:06.939052 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:06.939639 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:06.939646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:06.939652 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:06.940040 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:06.940212 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:06.941139 (+ 927us) server_negotiation.cc:299] Negotiation successful
0926 02:20:06.941411 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":257,"threads_started":1}
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
I20260926 02:20:06.942615 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:06.919551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34991 (local address 127.12.244.194:56825)
0926 02:20:06.920465 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:06.920519 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:06.922642 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:06.923512 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:06.923527 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:06.924432 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:06.925323 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:06.925403 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.926493 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.926501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:06.928879 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:06.928895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:06.930125 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:06.930131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:06.930356 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:06.931190 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:06.931221 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:06.933759 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:06.938141 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:06.938150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:06.938167 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:06.938676 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:06.939234 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:06.939238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:06.939241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:06.939499 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:06.940248 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:06.940258 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:06.940647 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 02:20:06.940970 (+ 323us) client_negotiation.cc:240] Negotiation successful
0926 02:20:06.941417 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":297,"threads_started":1}
I20260926 02:20:06.944525 17221 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
I20260926 02:20:06.944881 17221 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:06.945726 17221 heartbeater.cc:511] Master 127.12.244.254:34991 requested a full tablet report, sending...
I20260926 02:20:06.947552 16898 ts_manager.cc:194] Registered new tserver with Master: 2559d44a9b604f93947846787203e8a7 (127.12.244.194:46425)
I20260926 02:20:06.948510 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:56825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:20:06.970283 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34991
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:32789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:07.318451 17227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:07.318796 17227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:07.318984 17227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:07.326539 17227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:07.326689 17227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:07.326908 17227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:07.337241 17227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:32789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:07.338685 17227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:07.340197 17227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:07.354804 17232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:07.354771 17235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:07.354830 17233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:07.356310 17227 server_base.cc:1034] running on GCE node
I20260926 02:20:07.357033 17227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:07.358187 17227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:07.359437 17227 hybrid_clock.cc:648] HybridClock initialized: now 1790389207359395 us; error 63 us; skew 500 ppm
I20260926 02:20:07.359805 17227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:07 dist-test-slave-qfmw krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:07.363534 17227 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:20:07.365226 17227 webserver.cc:460] Webserver started at http://127.12.244.195:40027/ using document root <none> and password file <none>
I20260926 02:20:07.365962 17227 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:07.366072 17227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:07.366349 17227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:07.369999 17227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "391ee19564ef40c8946b01b30b909ad0"
format_stamp: "Formatted at 2026-09-26 02:20:07 on dist-test-slave-qfmw"
server_key: "019f266ab63873e4527bbe180b3950fd"
server_key_iv: "af92729edd91cfd53bb83337f6598cfb"
server_key_version: "encryptionkey@0"
I20260926 02:20:07.370834 17227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "391ee19564ef40c8946b01b30b909ad0"
format_stamp: "Formatted at 2026-09-26 02:20:07 on dist-test-slave-qfmw"
server_key: "019f266ab63873e4527bbe180b3950fd"
server_key_iv: "af92729edd91cfd53bb83337f6598cfb"
server_key_version: "encryptionkey@0"
I20260926 02:20:07.376778 17227 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 02:20:07.380828 17242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:07.382751 17227 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 02:20:07.382957 17227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "391ee19564ef40c8946b01b30b909ad0"
format_stamp: "Formatted at 2026-09-26 02:20:07 on dist-test-slave-qfmw"
server_key: "019f266ab63873e4527bbe180b3950fd"
server_key_iv: "af92729edd91cfd53bb83337f6598cfb"
server_key_version: "encryptionkey@0"
I20260926 02:20:07.383236 17227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:07.411832 17227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:07.416590 17227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:07.416849 17227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:07.417541 17087 heartbeater.cc:503] Master 127.12.244.254:34991 was elected leader, sending a full tablet report...
I20260926 02:20:07.418022 17227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:07.419720 17227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:07.419836 17227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:07.419979 17227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:07.420053 17227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:07.454789 17227 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:39341
I20260926 02:20:07.454835 17354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:39341 every 8 connection(s)
I20260926 02:20:07.456161 17227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:07.462284 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 17227
I20260926 02:20:07.462487 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:07.463142 13267 external_mini_cluster.cc:1428] Setting key 2bb50c409c1259ce7851943221137ad7
Sep 26 02:20:07 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389207, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:07.476922 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:07.458757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:34241 (local address 127.12.244.254:34991)
0926 02:20:07.459272 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:07.459277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:07.460275 (+ 998us) server_negotiation.cc:407] Connection header received
0926 02:20:07.462245 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:07.462250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:07.462380 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:07.462494 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:07.464285 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.465103 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.466687 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.467170 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.467302 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:07.470786 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:07.470791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:07.470795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:07.470826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:07.473505 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.474338 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.474344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.474349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.474482 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.474966 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.474973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.474980 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.475338 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:07.475510 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:07.476274 (+ 764us) server_negotiation.cc:299] Negotiation successful
0926 02:20:07.476520 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":222,"threads_started":1}
I20260926 02:20:07.478173 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:07.459377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34991 (local address 127.12.244.195:34241)
0926 02:20:07.460097 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:07.460142 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:07.461954 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:07.462765 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:07.462779 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:07.463515 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:07.464106 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:07.464130 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.465254 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.465260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:07.466535 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:07.466548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.467396 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.467405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:07.467653 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:07.468309 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:07.468333 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:07.470557 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:07.473679 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:07.473689 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:07.473710 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:07.474181 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:07.474610 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:07.474615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:07.474619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:07.474828 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:07.475475 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:07.475483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:07.475868 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 02:20:07.476166 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 02:20:07.476730 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":260,"threads_started":1}
I20260926 02:20:07.479630 17355 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34991
I20260926 02:20:07.479965 17355 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:07.480765 17355 heartbeater.cc:511] Master 127.12.244.254:34991 requested a full tablet report, sending...
I20260926 02:20:07.482362 16898 ts_manager.cc:194] Registered new tserver with Master: 391ee19564ef40c8946b01b30b909ad0 (127.12.244.195:39341)
I20260926 02:20:07.483268 16898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:34241
I20260926 02:20:07.491477 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:20:07 dist-test-slave-qfmw krb5kdc[16853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:20:07 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:20:07.542809 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:07.526541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.12.244.193:40949)
0926 02:20:07.527242 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:07.527250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:07.527280 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:20:07.527343 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:07.527351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:07.527570 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:07.527717 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:07.529090 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.530126 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.531437 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.532037 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.532146 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:07.536230 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:07.536247 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:07.536263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:07.536305 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:07.539470 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.540462 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.540475 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.540479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.540643 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.541175 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.541182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.541187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.541679 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:07.541996 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:07.542310 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 02:20:07.542516 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":335,"threads_started":1}
Sep 26 02:20:07 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:07.561831 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:07.547619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.12.244.254:34991)
0926 02:20:07.547820 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:07.547826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:07.548104 (+ 278us) server_negotiation.cc:407] Connection header received
0926 02:20:07.548246 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:07.548251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:07.548355 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:07.548466 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:07.549809 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.550565 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.551919 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.552324 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.552441 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:07.556588 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:07.556595 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:07.556598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:07.556637 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:07.559418 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.560062 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.560066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.560068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.560192 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.560576 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.560580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.560582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.560933 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:07.561115 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:07.561425 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 02:20:07.561597 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 02:20:07.562427 16898 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59046
I20260926 02:20:07.564174 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16959
I20260926 02:20:07.577250 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:07.565144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.12.244.193:40949)
0926 02:20:07.565324 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:07.565366 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:07.565553 (+ 187us) server_negotiation.cc:407] Connection header received
0926 02:20:07.565660 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:07.565664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:07.565754 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:07.565838 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:07.567045 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.567811 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.569015 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.569417 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.569517 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:07.572060 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:07.572065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:07.572069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:07.572104 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:07.574687 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.575504 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.575508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.575510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.575633 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.575980 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.575984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.575986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.576237 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:07.576531 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:07.576814 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 02:20:07.577018 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:20:07.577865 17082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:07.662966 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16959
I20260926 02:20:07.679742 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 17093
Sep 26 02:20:07 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:20:07.697723 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:07.680767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.12.244.194:46425)
0926 02:20:07.681470 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:07.681477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:07.681505 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:20:07.681563 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:07.681569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:07.681791 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:07.682006 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:07.683242 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.684279 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.685819 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.686454 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.686565 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:07.691296 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:07.691312 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:07.691329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:07.691381 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:07.694361 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.695279 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.695293 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.695299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.695490 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.695982 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.695986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.695988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.696339 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:07.696602 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:07.697114 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 02:20:07.697431 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":379,"threads_started":1}
I20260926 02:20:07.699148 17216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:07.790517 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 17093
I20260926 02:20:07.806239 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 17227
Sep 26 02:20:07 dist-test-slave-qfmw krb5kdc[16853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:20:07.825276 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:07.807329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.12.244.195:39341)
0926 02:20:07.808017 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:07.808023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:07.808055 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:20:07.808112 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:07.808118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:07.808348 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:07.808534 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:07.809843 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.811328 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.812916 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:07.813491 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:07.813647 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:07.818347 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:07.818363 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:07.818380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:07.818429 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:07.821849 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.822750 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.822767 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.822770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.822920 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:07.823440 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:07.823445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:07.823447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:07.823851 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:07.824217 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:07.824560 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 02:20:07.824885 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260926 02:20:07.826730 17350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:07.910709 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 17227
I20260926 02:20:07.926458 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 16866
I20260926 02:20:07.927776 16928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:08.031126 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 16866
2026-09-26T02:20:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4077 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:08 dist-test-slave-qfmw krb5kdc[17375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:08 dist-test-slave-qfmw krb5kdc[17375](info): set up 2 sockets
Sep 26 02:20:08 dist-test-slave-qfmw krb5kdc[17375](info): commencing operation
krb5kdc: starting...
W20260926 02:20:09.150393 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:09 dist-test-slave-qfmw krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:20:09.361269 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46335
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:46335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:09.707070 17388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:09.707501 17388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:09.707675 17388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:09.715358 17388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:09.715556 17388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:09.715677 17388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:09.715781 17388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:09.715894 17388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:09.726272 17388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:46335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:09.727720 17388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:09.729112 17388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:09.741870 17396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:09.741915 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:09.741962 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:09.743186 17388 server_base.cc:1034] running on GCE node
I20260926 02:20:09.744115 17388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:09.745640 17388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:09.746944 17388 hybrid_clock.cc:648] HybridClock initialized: now 1790389209746899 us; error 35 us; skew 500 ppm
I20260926 02:20:09.747378 17388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:09 dist-test-slave-qfmw krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389209, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:09.750780 17388 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:20:09.752207 17388 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:09.752351 17388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:09.752738 17388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:09.756067 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "5cac50a226cc4880ae5f8f7c0c0888f2"
format_stamp: "Formatted at 2026-09-26 02:20:09 on dist-test-slave-qfmw"
server_key: "55b43e9053437535afa9b3a2a432e974"
server_key_iv: "50519747161e55c1fee4fe82f26d87cc"
server_key_version: "encryptionkey@0"
I20260926 02:20:09.756958 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "5cac50a226cc4880ae5f8f7c0c0888f2"
format_stamp: "Formatted at 2026-09-26 02:20:09 on dist-test-slave-qfmw"
server_key: "55b43e9053437535afa9b3a2a432e974"
server_key_iv: "50519747161e55c1fee4fe82f26d87cc"
server_key_version: "encryptionkey@0"
I20260926 02:20:09.762506 17388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:20:09.766379 17402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:09.768016 17388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:20:09.768239 17388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "5cac50a226cc4880ae5f8f7c0c0888f2"
format_stamp: "Formatted at 2026-09-26 02:20:09 on dist-test-slave-qfmw"
server_key: "55b43e9053437535afa9b3a2a432e974"
server_key_iv: "50519747161e55c1fee4fe82f26d87cc"
server_key_version: "encryptionkey@0"
I20260926 02:20:09.768494 17388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:09.791210 17388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:09.796236 17388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:09.796547 17388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:09.815850 17388 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:46335
I20260926 02:20:09.815918 17453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:46335 every 8 connection(s)
I20260926 02:20:09.816836 17388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:09.820631 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:09.822646 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 17388
I20260926 02:20:09.822894 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:09.823606 13267 external_mini_cluster.cc:1428] Setting key 7f9e14ba79695f1f858399888e18c35e
I20260926 02:20:09.833796 17454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2: Bootstrap starting.
I20260926 02:20:09.837462 17454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:09.838665 17454 log.cc:824] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:20:09 dist-test-slave-qfmw krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:09.842468 17454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2: No bootstrap required, opened a new log
I20260926 02:20:09.850625 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:09.825995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.12.244.254:46335)
0926 02:20:09.827271 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:09.827285 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:09.827335 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:20:09.829213 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:09.829247 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:09.829743 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:09.830249 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:09.832080 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:09.833490 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:09.834931 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:09.835800 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:09.836055 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:09.841329 (+ 5274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:09.841351 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:09.841372 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:09.841426 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:09.844914 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:09.845967 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:09.845975 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:09.845983 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:09.846184 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:09.846652 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:09.846660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:09.846662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:09.847553 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:09.847968 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:09.848611 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 02:20:09.849173 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":615,"threads_started":1}
I20260926 02:20:09.852536 17454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46335 } }
I20260926 02:20:09.852962 17454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:09.853075 17454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cac50a226cc4880ae5f8f7c0c0888f2, State: Initialized, Role: FOLLOWER
I20260926 02:20:09.853718 17454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46335 } }
I20260926 02:20:09.853996 17454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:09.854190 17454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:09.854405 17454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:09.856762 17454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46335 } }
I20260926 02:20:09.857353 17454 leader_election.cc:302] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cac50a226cc4880ae5f8f7c0c0888f2; no voters:
I20260926 02:20:09.857975 17454 leader_election.cc:288] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:09.858143 17459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:09.859411 17459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5cac50a226cc4880ae5f8f7c0c0888f2, State: Running, Role: LEADER
I20260926 02:20:09.859998 17459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46335 } }
I20260926 02:20:09.860462 17454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:09.863303 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cac50a226cc4880ae5f8f7c0c0888f2. Latest consensus state: current_term: 1 leader_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46335 } } }
I20260926 02:20:09.863739 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:09.863278 17460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cac50a226cc4880ae5f8f7c0c0888f2" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46335 } } }
I20260926 02:20:09.863862 17460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:09.865795 17466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:09.871624 17466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:09.881919 17466 catalog_manager.cc:1347] Generated new cluster ID: 02f8cb73d36f48a591301c0c0482f8a3
I20260926 02:20:09.882071 17466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:09.900629 17466 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:09.902074 17466 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:09.911994 17466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2: Generated new TSK 0
I20260926 02:20:09.913028 17466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:20:09.972787 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46335
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:10.333690 17480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:10.334055 17480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:10.334163 17480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:10.342549 17480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:10.342692 17480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:10.342844 17480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:10.353521 17480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:10.354969 17480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:10.356485 17480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:10.371050 17488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:10.371033 17485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:10.371083 17486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:10.372653 17480 server_base.cc:1034] running on GCE node
I20260926 02:20:10.373492 17480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:10.374716 17480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:10.376000 17480 hybrid_clock.cc:648] HybridClock initialized: now 1790389210375956 us; error 61 us; skew 500 ppm
I20260926 02:20:10.376452 17480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:10 dist-test-slave-qfmw krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:10.380116 17480 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:20:10.381448 17480 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:10.381596 17480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:10.381942 17480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:10.385202 17480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "8c455f9be8be4911aa0321cb5f6a3098"
format_stamp: "Formatted at 2026-09-26 02:20:10 on dist-test-slave-qfmw"
server_key: "04c2dbf7768a00138cb711c324aef8fd"
server_key_iv: "c6716f4bfb1760a910b9c99fa8c14db5"
server_key_version: "encryptionkey@0"
I20260926 02:20:10.386122 17480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "8c455f9be8be4911aa0321cb5f6a3098"
format_stamp: "Formatted at 2026-09-26 02:20:10 on dist-test-slave-qfmw"
server_key: "04c2dbf7768a00138cb711c324aef8fd"
server_key_iv: "c6716f4bfb1760a910b9c99fa8c14db5"
server_key_version: "encryptionkey@0"
I20260926 02:20:10.392057 17480 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:20:10.396239 17494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:10.397967 17480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:20:10.398203 17480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "8c455f9be8be4911aa0321cb5f6a3098"
format_stamp: "Formatted at 2026-09-26 02:20:10 on dist-test-slave-qfmw"
server_key: "04c2dbf7768a00138cb711c324aef8fd"
server_key_iv: "c6716f4bfb1760a910b9c99fa8c14db5"
server_key_version: "encryptionkey@0"
I20260926 02:20:10.398471 17480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:10.415071 17480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:10.420135 17480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:10.420451 17480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:10.421505 17480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:10.422984 17480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:10.423146 17480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:10.423278 17480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:10.423375 17480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:10.457866 17480 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:42331
I20260926 02:20:10.457922 17606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:42331 every 8 connection(s)
I20260926 02:20:10.458770 17480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:10.465226 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 17480
I20260926 02:20:10.465492 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:10.466146 13267 external_mini_cluster.cc:1428] Setting key 2ee8f1dd5ca02a39a69d3be90e84d2d7
Sep 26 02:20:10 dist-test-slave-qfmw krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:10.482728 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:10.461348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:33209 (local address 127.12.244.254:46335)
0926 02:20:10.461811 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:10.461816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:10.462833 (+ 1017us) server_negotiation.cc:407] Connection header received
0926 02:20:10.464853 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:10.464859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:10.464962 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:10.465087 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:10.467031 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:10.468153 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:10.469955 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:10.470706 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:10.470877 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:10.475685 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:10.475693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:10.475705 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:10.475753 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:10.478827 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:10.480003 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:10.480009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:10.480011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:10.480145 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:10.480695 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:10.480699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:10.480701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:10.481004 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:10.481128 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:10.482158 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0926 02:20:10.482341 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":263,"threads_started":1}
I20260926 02:20:10.484475 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:10.461949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46335 (local address 127.12.244.193:33209)
0926 02:20:10.462652 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:10.462698 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:10.464513 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:10.465349 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:10.465359 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:10.466033 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:10.466766 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:10.466793 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:10.468349 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:10.468353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:10.469769 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:10.469783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:10.471061 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:10.471070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:10.471412 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:10.472196 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:10.472227 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:10.475359 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:10.479033 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:10.479046 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:10.479070 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:10.479831 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:10.480286 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:10.480293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:10.480300 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:10.480571 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:10.481151 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:10.481163 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:10.481712 (+ 549us) client_negotiation.cc:769] Sending connection context
0926 02:20:10.482055 (+ 343us) client_negotiation.cc:240] Negotiation successful
0926 02:20:10.482727 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":272,"threads_started":1}
I20260926 02:20:10.486466 17607 heartbeater.cc:347] Connected to a master server at 127.12.244.254:46335
I20260926 02:20:10.486833 17607 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:20:10.487732 17607 heartbeater.cc:511] Master 127.12.244.254:46335 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:20:10.490168 17419 ts_manager.cc:194] Registered new tserver with Master: 8c455f9be8be4911aa0321cb5f6a3098 (127.12.244.193:42331)
I20260926 02:20:10.491854 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:33209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:20:10.515365 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46335
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:10.873832 17613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:10.874219 17613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:10.874459 17613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:10.882122 17613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:10.882273 17613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:10.882500 17613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:10.892812 17613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:10.894352 17613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:10.896108 17613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:10.911630 17619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:10.911662 17621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:10.911593 17618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:10.912575 17613 server_base.cc:1034] running on GCE node
I20260926 02:20:10.913249 17613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:10.914291 17613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:10.915515 17613 hybrid_clock.cc:648] HybridClock initialized: now 1790389210915455 us; error 55 us; skew 500 ppm
I20260926 02:20:10.915851 17613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:10 dist-test-slave-qfmw krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:10.919449 17613 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:20:10.920796 17613 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:10.920941 17613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:10.921274 17613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:10.924456 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0"
format_stamp: "Formatted at 2026-09-26 02:20:10 on dist-test-slave-qfmw"
server_key: "8340ba6f373795aa3868b4991b240fe6"
server_key_iv: "307f46bad448fb29cfddb0bbb83dea5a"
server_key_version: "encryptionkey@0"
I20260926 02:20:10.925426 17613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0"
format_stamp: "Formatted at 2026-09-26 02:20:10 on dist-test-slave-qfmw"
server_key: "8340ba6f373795aa3868b4991b240fe6"
server_key_iv: "307f46bad448fb29cfddb0bbb83dea5a"
server_key_version: "encryptionkey@0"
I20260926 02:20:10.931174 17613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:20:10.935298 17627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:10.936973 17613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:20:10.937209 17613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0"
format_stamp: "Formatted at 2026-09-26 02:20:10 on dist-test-slave-qfmw"
server_key: "8340ba6f373795aa3868b4991b240fe6"
server_key_iv: "307f46bad448fb29cfddb0bbb83dea5a"
server_key_version: "encryptionkey@0"
I20260926 02:20:10.937470 17613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:10.952240 17613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:10.957273 17613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:10.957634 17613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:10.958698 17613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:10.960325 17613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:10.960457 17613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:10.960582 17613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:10.960665 17613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:10.995805 17613 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:40003
I20260926 02:20:10.995937 17739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:40003 every 8 connection(s)
I20260926 02:20:10.996834 17613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:11.007170 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 17613
I20260926 02:20:11.007427 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:11.007920 13267 external_mini_cluster.cc:1428] Setting key a96a90451d1dbf8012429eb3310e25cc
Sep 26 02:20:11 dist-test-slave-qfmw krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389210, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:11.020642 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:10.999502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:41813 (local address 127.12.244.254:46335)
0926 02:20:10.999936 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:10.999943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.001021 (+ 1078us) server_negotiation.cc:407] Connection header received
0926 02:20:11.003050 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.003056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.003202 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.003316 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:11.005116 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.005929 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.008122 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.008749 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.008923 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.012893 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:11.012903 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:11.012911 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:11.012963 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:11.016663 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:11.017956 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:11.017962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:11.017964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:11.018110 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:11.018660 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:11.018663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:11.018665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:11.018941 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:11.019058 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.020156 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.020366 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":205,"threads_started":1}
I20260926 02:20:11.022257 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.000116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46335 (local address 127.12.244.194:41813)
0926 02:20:11.000860 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:11.000899 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:11.002580 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:11.003607 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:11.003616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:11.004246 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:11.004886 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.004905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.006166 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.006171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.007887 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.007906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.009034 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.009042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.009299 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.010124 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:11.010151 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:11.012593 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:11.016872 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:11.016885 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:11.016909 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:11.017723 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:11.018241 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:11.018249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:11.018255 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:11.018546 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:11.019078 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:11.019118 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:11.019664 (+ 546us) client_negotiation.cc:769] Sending connection context
0926 02:20:11.020026 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 02:20:11.020656 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":294,"threads_started":1}
I20260926 02:20:11.024111 17740 heartbeater.cc:347] Connected to a master server at 127.12.244.254:46335
I20260926 02:20:11.024444 17740 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:11.025168 17740 heartbeater.cc:511] Master 127.12.244.254:46335 requested a full tablet report, sending...
I20260926 02:20:11.026937 17419 ts_manager.cc:194] Registered new tserver with Master: 62318a12b2ca4c3c92927bc2ed5c1ec0 (127.12.244.194:40003)
I20260926 02:20:11.028266 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:41813
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:20:11.059484 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46335
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:11.407426 17746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:11.407778 17746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:11.407930 17746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:11.415573 17746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:11.415709 17746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:11.415906 17746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:11.426146 17746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:11.428012 17746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:11.429436 17746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:11.444151 17752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:11.444183 17754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:11.444222 17751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:11.448931 17746 server_base.cc:1034] running on GCE node
I20260926 02:20:11.449676 17746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:11.450778 17746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:11.452020 17746 hybrid_clock.cc:648] HybridClock initialized: now 1790389211451925 us; error 99 us; skew 500 ppm
I20260926 02:20:11.452378 17746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:11 dist-test-slave-qfmw krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389211, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:11.455859 17746 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:20:11.457353 17746 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:11.457489 17746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:11.457876 17746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:11.462095 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "084740f2dd3f44b6b227c5e7124ca9bd"
format_stamp: "Formatted at 2026-09-26 02:20:11 on dist-test-slave-qfmw"
server_key: "8e2337a371765984ee0c196a7558d784"
server_key_iv: "b5d65f367dd4778292d16df29e20f4f8"
server_key_version: "encryptionkey@0"
I20260926 02:20:11.463201 17746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "084740f2dd3f44b6b227c5e7124ca9bd"
format_stamp: "Formatted at 2026-09-26 02:20:11 on dist-test-slave-qfmw"
server_key: "8e2337a371765984ee0c196a7558d784"
server_key_iv: "b5d65f367dd4778292d16df29e20f4f8"
server_key_version: "encryptionkey@0"
I20260926 02:20:11.468958 17746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:20:11.472847 17760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:11.474581 17746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:20:11.474797 17746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "084740f2dd3f44b6b227c5e7124ca9bd"
format_stamp: "Formatted at 2026-09-26 02:20:11 on dist-test-slave-qfmw"
server_key: "8e2337a371765984ee0c196a7558d784"
server_key_iv: "b5d65f367dd4778292d16df29e20f4f8"
server_key_version: "encryptionkey@0"
I20260926 02:20:11.475032 17746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:11.488669 17746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:11.493187 17746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:11.493489 17746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:11.494511 17746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:11.495316 17607 heartbeater.cc:503] Master 127.12.244.254:46335 was elected leader, sending a full tablet report...
I20260926 02:20:11.496196 17746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:11.496316 17746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:11.496454 17746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:11.496541 17746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:11.529938 17746 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:44071
I20260926 02:20:11.530061 17872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:44071 every 8 connection(s)
I20260926 02:20:11.530972 17746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:11.541090 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 17746
I20260926 02:20:11.541321 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:11.541747 13267 external_mini_cluster.cc:1428] Setting key a4091d895b5c73aec42633405f72fdae
Sep 26 02:20:11 dist-test-slave-qfmw krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389211, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:11.552520 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.533836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:37469 (local address 127.12.244.254:46335)
0926 02:20:11.534305 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:11.534309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.535337 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 02:20:11.537223 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.537229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.537322 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.537448 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:11.539025 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.540031 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.541870 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.542456 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.542591 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.546665 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:11.546670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:11.546673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:11.546707 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:11.549294 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:11.550119 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:11.550123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:11.550126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:11.550253 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:11.550656 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:11.550659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:11.550661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:11.550935 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:11.551056 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.551886 (+ 830us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.552124 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":240,"threads_started":1}
I20260926 02:20:11.553434 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.534374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46335 (local address 127.12.244.195:37469)
0926 02:20:11.535166 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:11.535217 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:11.536926 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:11.537689 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:11.537698 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:11.538240 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:11.538835 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.538852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.540193 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.540197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.541709 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.541722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.542728 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.542736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.543023 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.543951 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:11.543989 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:11.546453 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:11.549486 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:11.549494 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:11.549511 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:11.549992 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:11.550356 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:11.550360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:11.550362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:11.550561 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:11.551084 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:11.551147 (+ 63us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:11.551542 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 02:20:11.551806 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 02:20:11.552216 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":285,"threads_started":1}
I20260926 02:20:11.555119 17873 heartbeater.cc:347] Connected to a master server at 127.12.244.254:46335
I20260926 02:20:11.555457 17873 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:11.556402 17873 heartbeater.cc:511] Master 127.12.244.254:46335 requested a full tablet report, sending...
I20260926 02:20:11.558473 17419 ts_manager.cc:194] Registered new tserver with Master: 084740f2dd3f44b6b227c5e7124ca9bd (127.12.244.195:44071)
I20260926 02:20:11.559345 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:37469
I20260926 02:20:11.570474 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:20:11.589177 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.575823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.12.244.254:46335)
0926 02:20:11.576000 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:11.576005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.576425 (+ 420us) server_negotiation.cc:407] Connection header received
0926 02:20:11.576541 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.576545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.576614 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.576706 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:11.578179 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.579019 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.580393 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.580788 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.580927 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.583415 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:11.583420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:11.583423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:11.583461 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:11.586297 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:11.587027 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:11.587033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:11.587037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:11.587231 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:11.587642 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:11.587648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:11.587652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:11.587995 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:11.588252 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.588600 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.588837 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:20:11.593627 17419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:20:11.600723 17419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:11.625790 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.614300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:40003 (local address 127.0.0.1:43992)
0926 02:20:11.615170 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:11.615213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:11.615428 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:11.617529 (+ 2101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:11.617540 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:11.617573 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:11.618082 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.618105 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.620125 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.620131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.622710 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.622722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.624651 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.624658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.624866 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.624938 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 02:20:11.625070 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 02:20:11.625282 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":53,"thread_start_us":380,"threads_started":1}
I20260926 02:20:11.625986 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.616173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.12.244.193:42331)
0926 02:20:11.617323 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:11.617335 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.617986 (+ 651us) server_negotiation.cc:407] Connection header received
0926 02:20:11.618274 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.618282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.618555 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.618721 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:11.619358 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.620567 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.623465 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.624415 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.624655 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.624870 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.625437 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.625643 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":782,"threads_started":1}
I20260926 02:20:11.626595 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.615323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.12.244.195:44071)
0926 02:20:11.616097 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:11.616117 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.616864 (+ 747us) server_negotiation.cc:407] Connection header received
0926 02:20:11.616980 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.616988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.617265 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.617473 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:11.618766 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.620266 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.623411 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.624846 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.625047 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.625212 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.626062 (+ 850us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.626225 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":431,"threads_started":1}
I20260926 02:20:11.626828 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.616549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:42331 (local address 127.0.0.1:41082)
0926 02:20:11.617855 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:11.617872 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:11.618090 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:11.618792 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:11.618799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:11.618823 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:11.619191 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.619200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.620705 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.620710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.623229 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.623243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.625024 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.625030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.625205 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.625235 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:20:11.626259 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0926 02:20:11.626389 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"mutex_wait_us":75,"thread_start_us":998,"threads_started":1}
I20260926 02:20:11.627028 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.614555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.12.244.194:40003)
0926 02:20:11.616887 (+ 2332us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:11.616896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.616918 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:20:11.616994 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.617000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.617265 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.618040 (+ 775us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:11.618271 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.619976 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.622887 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.624385 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.626257 (+ 1872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.626447 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.626520 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.626693 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2153,"thread_start_us":2048,"threads_started":1}
I20260926 02:20:11.627305 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.614724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44071 (local address 127.0.0.1:34018)
0926 02:20:11.616585 (+ 1861us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:11.616607 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:11.616736 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:11.617764 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:11.617768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:11.617790 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:11.618552 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.618564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.620404 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.620410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.623238 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.623247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.625365 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.625371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.625559 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.625590 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:20:11.625687 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:20:11.625848 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"mutex_wait_us":211,"thread_start_us":1566,"threads_started":1}
I20260926 02:20:11.630774 17542 tablet_service.cc:1467] Processing CreateTablet for tablet ea2ac2c45216428dbd35533818d88124 (DEFAULT_TABLE table=test-table [id=a7a6d9b3348346b7b2bebf1c38a1e282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:11.630770 17675 tablet_service.cc:1467] Processing CreateTablet for tablet ea2ac2c45216428dbd35533818d88124 (DEFAULT_TABLE table=test-table [id=a7a6d9b3348346b7b2bebf1c38a1e282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:11.630749 17808 tablet_service.cc:1467] Processing CreateTablet for tablet ea2ac2c45216428dbd35533818d88124 (DEFAULT_TABLE table=test-table [id=a7a6d9b3348346b7b2bebf1c38a1e282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:11.632114 17808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2ac2c45216428dbd35533818d88124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:11.632167 17542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2ac2c45216428dbd35533818d88124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:11.632169 17675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2ac2c45216428dbd35533818d88124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:11.645042 17892 tablet_bootstrap.cc:492] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Bootstrap starting.
I20260926 02:20:11.647348 17893 tablet_bootstrap.cc:492] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Bootstrap starting.
I20260926 02:20:11.648821 17894 tablet_bootstrap.cc:492] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Bootstrap starting.
I20260926 02:20:11.650836 17892 tablet_bootstrap.cc:654] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:11.651547 17893 tablet_bootstrap.cc:654] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:11.651739 17894 tablet_bootstrap.cc:654] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:11.652390 17892 log.cc:824] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:11.652871 17894 log.cc:824] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:11.652899 17893 log.cc:824] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:11.656253 17894 tablet_bootstrap.cc:492] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: No bootstrap required, opened a new log
I20260926 02:20:11.656585 17894 ts_tablet_manager.cc:1397] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:20:11.656658 17892 tablet_bootstrap.cc:492] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: No bootstrap required, opened a new log
I20260926 02:20:11.656658 17893 tablet_bootstrap.cc:492] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: No bootstrap required, opened a new log
I20260926 02:20:11.656930 17892 ts_tablet_manager.cc:1397] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20260926 02:20:11.656939 17893 ts_tablet_manager.cc:1397] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:20:11.664605 17894 raft_consensus.cc:348] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.665058 17894 raft_consensus.cc:374] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:11.665158 17894 raft_consensus.cc:729] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c455f9be8be4911aa0321cb5f6a3098, State: Initialized, Role: FOLLOWER
I20260926 02:20:11.665853 17894 consensus_queue.cc:260] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.667856 17894 ts_tablet_manager.cc:1428] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 02:20:11.668222 17893 raft_consensus.cc:348] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.668222 17892 raft_consensus.cc:348] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.668691 17893 raft_consensus.cc:374] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:11.668691 17892 raft_consensus.cc:374] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:11.668848 17892 raft_consensus.cc:729] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084740f2dd3f44b6b227c5e7124ca9bd, State: Initialized, Role: FOLLOWER
I20260926 02:20:11.668848 17893 raft_consensus.cc:729] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62318a12b2ca4c3c92927bc2ed5c1ec0, State: Initialized, Role: FOLLOWER
I20260926 02:20:11.669540 17892 consensus_queue.cc:260] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.669540 17893 consensus_queue.cc:260] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.671332 17740 heartbeater.cc:503] Master 127.12.244.254:46335 was elected leader, sending a full tablet report...
I20260926 02:20:11.671603 17892 ts_tablet_manager.cc:1428] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260926 02:20:11.671878 17893 ts_tablet_manager.cc:1428] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 02:20:11.672513 17873 heartbeater.cc:503] Master 127.12.244.254:46335 was elected leader, sending a full tablet report...
W20260926 02:20:11.711268 17608 tablet.cc:2369] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:20:11.748833 17741 tablet.cc:2369] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:20:11.782649 17874 tablet.cc:2369] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:20:11.979523 17899 raft_consensus.cc:482] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:11.979938 17899 raft_consensus.cc:504] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.981666 17899 leader_election.cc:288] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62318a12b2ca4c3c92927bc2ed5c1ec0 (127.12.244.194:40003), 8c455f9be8be4911aa0321cb5f6a3098 (127.12.244.193:42331)
I20260926 02:20:11.988436 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.982131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:46923 (local address 127.12.244.194:40003)
0926 02:20:11.982338 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:11.982345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.982471 (+ 126us) server_negotiation.cc:407] Connection header received
0926 02:20:11.982696 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.982704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.982804 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.982920 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:11.983512 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.984502 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.986261 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.987554 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.987717 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.987796 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.988061 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.988173 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:20:11.988497 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.982077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:40003 (local address 127.12.244.195:46923)
0926 02:20:11.982310 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:11.982332 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:11.982524 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:11.983030 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:11.983035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:11.983070 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:11.983345 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.983358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.984645 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.984650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.986018 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.986030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.987709 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.987713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.987854 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.987908 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:20:11.988016 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:20:11.988164 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":4}
I20260926 02:20:11.988782 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.982509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:42331 (local address 127.12.244.195:51445)
0926 02:20:11.983132 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:11.983156 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:11.983281 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:11.983617 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:11.983622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:11.983648 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:11.984008 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.984020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.984982 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.984988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.986393 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:11.986410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.988004 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.988011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:11.988170 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.988197 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 02:20:11.988280 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 02:20:11.988403 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":376,"threads_started":1}
I20260926 02:20:11.989169 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:11.982664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:51445 (local address 127.12.244.193:42331)
0926 02:20:11.982883 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:11.982891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:11.983259 (+ 368us) server_negotiation.cc:407] Connection header received
0926 02:20:11.983395 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:11.983432 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:11.983505 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:11.983577 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:11.984149 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.984824 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.986564 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:11.987807 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:11.987948 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:11.988035 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:11.988320 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 02:20:11.988453 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:20:11.989557 17695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2ac2c45216428dbd35533818d88124" candidate_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" is_pre_election: true
I20260926 02:20:11.990067 17695 raft_consensus.cc:2454] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 084740f2dd3f44b6b227c5e7124ca9bd in term 0.
I20260926 02:20:11.990044 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2ac2c45216428dbd35533818d88124" candidate_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c455f9be8be4911aa0321cb5f6a3098" is_pre_election: true
I20260926 02:20:11.990492 17562 raft_consensus.cc:2454] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 084740f2dd3f44b6b227c5e7124ca9bd in term 0.
I20260926 02:20:11.990777 17762 leader_election.cc:302] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084740f2dd3f44b6b227c5e7124ca9bd, 62318a12b2ca4c3c92927bc2ed5c1ec0; no voters:
I20260926 02:20:11.991211 17899 raft_consensus.cc:2781] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:11.991355 17899 raft_consensus.cc:482] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:11.991469 17899 raft_consensus.cc:3037] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:11.993642 17899 raft_consensus.cc:504] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:11.994297 17899 leader_election.cc:288] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [CANDIDATE]: Term 1 election: Requested vote from peers 62318a12b2ca4c3c92927bc2ed5c1ec0 (127.12.244.194:40003), 8c455f9be8be4911aa0321cb5f6a3098 (127.12.244.193:42331)
I20260926 02:20:11.994802 17695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2ac2c45216428dbd35533818d88124" candidate_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0"
I20260926 02:20:11.994889 17562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2ac2c45216428dbd35533818d88124" candidate_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c455f9be8be4911aa0321cb5f6a3098"
I20260926 02:20:11.995014 17695 raft_consensus.cc:3037] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:11.995208 17562 raft_consensus.cc:3037] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:11.997511 17695 raft_consensus.cc:2454] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 084740f2dd3f44b6b227c5e7124ca9bd in term 1.
I20260926 02:20:11.998025 17762 leader_election.cc:302] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084740f2dd3f44b6b227c5e7124ca9bd, 62318a12b2ca4c3c92927bc2ed5c1ec0; no voters:
I20260926 02:20:11.998325 17899 raft_consensus.cc:2781] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:11.998409 17562 raft_consensus.cc:2454] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 084740f2dd3f44b6b227c5e7124ca9bd in term 1.
I20260926 02:20:11.998853 17899 raft_consensus.cc:686] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 1 LEADER]: Becoming Leader. State: Replica: 084740f2dd3f44b6b227c5e7124ca9bd, State: Running, Role: LEADER
I20260926 02:20:11.999398 17899 consensus_queue.cc:237] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } }
I20260926 02:20:12.004520 17417 catalog_manager.cc:5495] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 084740f2dd3f44b6b227c5e7124ca9bd (127.12.244.195). New cstate: current_term: 1 leader_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "084740f2dd3f44b6b227c5e7124ca9bd" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44071 } health_report { overall_health: HEALTHY } } }
I20260926 02:20:12.088850 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:12.083249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.12.244.195:44071)
0926 02:20:12.083497 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:12.083504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:12.083606 (+ 102us) server_negotiation.cc:407] Connection header received
0926 02:20:12.083858 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:12.083863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:12.083941 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:12.084099 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:12.084486 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.085240 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.086593 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.087011 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.087162 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:12.087681 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:12.088204 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:12.088390 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:12.088459 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 02:20:12.088546 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:20:12.097412 17562 raft_consensus.cc:1261] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 1 FOLLOWER]: Refusing update from remote peer 084740f2dd3f44b6b227c5e7124ca9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:12.097440 17695 raft_consensus.cc:1261] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 1 FOLLOWER]: Refusing update from remote peer 084740f2dd3f44b6b227c5e7124ca9bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:12.098443 17902 consensus_queue.cc:1035] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c455f9be8be4911aa0321cb5f6a3098" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:12.098928 17899 consensus_queue.cc:1035] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "62318a12b2ca4c3c92927bc2ed5c1ec0" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:12.116550 17907 mvcc.cc:205] Tried to move back new op lower bound from 7333434212740460544 to 7333434212354211840. Current Snapshot: MvccSnapshot[applied={T|T < 7333434212740460544}]
I20260926 02:20:12.128515 17417 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:12.128780 17417 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:12.132717 17417 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cac50a226cc4880ae5f8f7c0c0888f2: Sending DeleteTablet for 3 replicas of tablet ea2ac2c45216428dbd35533818d88124
I20260926 02:20:12.134268 17542 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2ac2c45216428dbd35533818d88124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:12 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41082
I20260926 02:20:12.134258 17808 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2ac2c45216428dbd35533818d88124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:12 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:34018
I20260926 02:20:12.134744 17675 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2ac2c45216428dbd35533818d88124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:12 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:43992
I20260926 02:20:12.136483 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 17480
I20260926 02:20:12.150096 17914 tablet_replica.cc:330] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: stopping tablet replica
I20260926 02:20:12.150741 17914 raft_consensus.cc:2229] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:12.151448 17914 raft_consensus.cc:2258] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:12.152495 17915 tablet_replica.cc:330] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: stopping tablet replica
I20260926 02:20:12.153035 17915 raft_consensus.cc:2229] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:12.153450 17916 tablet_replica.cc:330] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: stopping tablet replica
I20260926 02:20:12.153746 17915 raft_consensus.cc:2258] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:12.153908 17916 raft_consensus.cc:2229] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:12.154395 17916 raft_consensus.cc:2258] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:12.156124 17915 ts_tablet_manager.cc:1905] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:12.156575 17916 ts_tablet_manager.cc:1905] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:12.162230 17914 ts_tablet_manager.cc:1905] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:12.163841 17916 ts_tablet_manager.cc:1918] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:12.164036 17916 log.cc:1196] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/ea2ac2c45216428dbd35533818d88124
I20260926 02:20:12.164626 17916 ts_tablet_manager.cc:1939] T ea2ac2c45216428dbd35533818d88124 P 62318a12b2ca4c3c92927bc2ed5c1ec0: Deleting consensus metadata
I20260926 02:20:12.166718 17404 catalog_manager.cc:4841] TS 62318a12b2ca4c3c92927bc2ed5c1ec0 (127.12.244.194:40003): tablet ea2ac2c45216428dbd35533818d88124 (table test-table [id=a7a6d9b3348346b7b2bebf1c38a1e282]) successfully deleted
Sep 26 02:20:12 dist-test-slave-qfmw krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:20:12.169312 17914 ts_tablet_manager.cc:1918] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:12.169502 17914 log.cc:1196] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/ea2ac2c45216428dbd35533818d88124
I20260926 02:20:12.170022 17914 ts_tablet_manager.cc:1939] T ea2ac2c45216428dbd35533818d88124 P 8c455f9be8be4911aa0321cb5f6a3098: Deleting consensus metadata
I20260926 02:20:12.171985 17915 ts_tablet_manager.cc:1918] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:12.172227 17915 log.cc:1196] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/ea2ac2c45216428dbd35533818d88124
I20260926 02:20:12.172264 17404 catalog_manager.cc:4841] TS 8c455f9be8be4911aa0321cb5f6a3098 (127.12.244.193:42331): tablet ea2ac2c45216428dbd35533818d88124 (table test-table [id=a7a6d9b3348346b7b2bebf1c38a1e282]) successfully deleted
I20260926 02:20:12.173123 17915 ts_tablet_manager.cc:1939] T ea2ac2c45216428dbd35533818d88124 P 084740f2dd3f44b6b227c5e7124ca9bd: Deleting consensus metadata
I20260926 02:20:12.175318 17405 catalog_manager.cc:4841] TS 084740f2dd3f44b6b227c5e7124ca9bd (127.12.244.195:44071): tablet ea2ac2c45216428dbd35533818d88124 (table test-table [id=a7a6d9b3348346b7b2bebf1c38a1e282]) successfully deleted
I20260926 02:20:12.176937 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:12.138361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.12.244.193:42331)
0926 02:20:12.138622 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:12.138631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:12.152769 (+ 14138us) server_negotiation.cc:407] Connection header received
0926 02:20:12.152998 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:12.153003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:12.153098 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:12.153219 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:12.155343 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.156469 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.158522 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.159196 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.159315 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:12.169005 (+ 9690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:12.169022 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:12.169044 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:12.169102 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:12.173515 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:12.174496 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:12.174504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:12.174506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:12.174649 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:12.175176 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:12.175179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:12.175182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:12.175588 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:12.175932 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:12.176283 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 02:20:12.176501 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:20:12.177567 17602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:12.276207 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 17480
I20260926 02:20:12.295439 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 17613
Sep 26 02:20:12 dist-test-slave-qfmw krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:20:12.312382 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:12.296389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.12.244.194:40003)
0926 02:20:12.296612 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:12.296618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:12.296642 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:20:12.296710 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:12.296717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:12.296798 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:12.296912 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:12.298217 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.299454 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.300678 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.301322 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.301457 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:12.305984 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:12.306004 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:12.306022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:12.306067 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:12.309318 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:12.310208 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:12.310216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:12.310218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:12.310360 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:12.310810 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:12.310814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:12.310816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:12.311343 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:12.311595 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:12.311902 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 02:20:12.312074 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:20:12.313055 17735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:12.404290 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 17613
I20260926 02:20:12.426663 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 17746
Sep 26 02:20:12 dist-test-slave-qfmw krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:20:12.444165 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:12.427733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.12.244.195:44071)
0926 02:20:12.427935 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:12.427943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:12.427964 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:20:12.428017 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:12.428021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:12.428080 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:12.428203 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:12.429559 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.430364 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.431752 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:12.432229 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:12.432376 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:12.437375 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:12.437391 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:12.437408 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:12.437491 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:12.441024 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:12.441947 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:12.441955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:12.441957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:12.442129 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:12.442605 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:12.442609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:12.442611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:12.443000 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:12.443338 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:12.443628 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 02:20:12.443815 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:20:12.445020 17868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:12.566833 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 17746
I20260926 02:20:12.582563 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 17388
I20260926 02:20:12.583818 17449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:12.695701 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 17388
2026-09-26T02:20:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4666 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T02:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:12Z Disabled control of system clock
I20260926 02:20:12.768158 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:40319
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:40319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:20:13.115332 17925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:13.115665 17925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:13.115749 17925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:13.123377 17925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:13.123505 17925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:13.123543 17925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:13.123592 17925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:13.123637 17925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:13.133797 17925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:40319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:40319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:13.135190 17925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:13.137043 17925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:13.151171 17933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:13.151250 17931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:13.151204 17930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:13.157465 17925 server_base.cc:1034] running on GCE node
I20260926 02:20:13.158525 17925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:13.160261 17925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:13.161532 17925 hybrid_clock.cc:648] HybridClock initialized: now 1790389213161491 us; error 53 us; skew 500 ppm
I20260926 02:20:13.162003 17925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:13.164851 17925 webserver.cc:460] Webserver started at http://127.12.244.254:42991/ using document root <none> and password file <none>
I20260926 02:20:13.165655 17925 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:13.165809 17925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:13.166183 17925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:13.169397 17925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "233e61279a444325ad8d7d41a7e45913"
format_stamp: "Formatted at 2026-09-26 02:20:13 on dist-test-slave-qfmw"
server_key: "531143fcfb8be8179996b018a3960e15"
server_key_iv: "c3da0f18cc549df5d62134a49c2f8799"
server_key_version: "encryptionkey@0"
I20260926 02:20:13.170301 17925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "233e61279a444325ad8d7d41a7e45913"
format_stamp: "Formatted at 2026-09-26 02:20:13 on dist-test-slave-qfmw"
server_key: "531143fcfb8be8179996b018a3960e15"
server_key_iv: "c3da0f18cc549df5d62134a49c2f8799"
server_key_version: "encryptionkey@0"
I20260926 02:20:13.176216 17925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:20:13.180310 17939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:13.181950 17925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:20:13.182178 17925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "233e61279a444325ad8d7d41a7e45913"
format_stamp: "Formatted at 2026-09-26 02:20:13 on dist-test-slave-qfmw"
server_key: "531143fcfb8be8179996b018a3960e15"
server_key_iv: "c3da0f18cc549df5d62134a49c2f8799"
server_key_version: "encryptionkey@0"
I20260926 02:20:13.182405 17925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:13.188795 17925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:13.189682 17925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:13.190024 17925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:13.208897 17925 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:40319
I20260926 02:20:13.208982 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:40319 every 8 connection(s)
I20260926 02:20:13.210356 17925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:13.214406 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:13.220346 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 17925
I20260926 02:20:13.220543 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:13.221349 13267 external_mini_cluster.cc:1428] Setting key 793b69d6d1a1c23db3bc9a3289bc243f
I20260926 02:20:13.226922 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913: Bootstrap starting.
I20260926 02:20:13.230603 17991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:13.230775 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:13.223716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.12.244.254:40319)
0926 02:20:13.225019 (+ 1303us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:13.225038 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:13.225085 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:20:13.226824 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:13.226850 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:13.227413 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:13.227840 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:13.229010 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:13.229016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:13.229030 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:13.229043 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:13.229110 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:13.229126 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:13.229355 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:13.229500 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:20:13.229772 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":451,"threads_started":1}
I20260926 02:20:13.232239 17991 log.cc:824] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:13.235443 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913: No bootstrap required, opened a new log
I20260926 02:20:13.242983 17991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e61279a444325ad8d7d41a7e45913" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 40319 } }
I20260926 02:20:13.243446 17991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:13.243574 17991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233e61279a444325ad8d7d41a7e45913, State: Initialized, Role: FOLLOWER
I20260926 02:20:13.244275 17991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e61279a444325ad8d7d41a7e45913" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 40319 } }
I20260926 02:20:13.244529 17991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:13.244671 17991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:13.244854 17991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:13.247753 17991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e61279a444325ad8d7d41a7e45913" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 40319 } }
I20260926 02:20:13.248261 17991 leader_election.cc:302] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 233e61279a444325ad8d7d41a7e45913; no voters:
I20260926 02:20:13.248843 17991 leader_election.cc:288] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:13.249024 17996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:13.249818 17996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [term 1 LEADER]: Becoming Leader. State: Replica: 233e61279a444325ad8d7d41a7e45913, State: Running, Role: LEADER
I20260926 02:20:13.250391 17996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e61279a444325ad8d7d41a7e45913" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 40319 } }
I20260926 02:20:13.251417 17991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:13.254047 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 233e61279a444325ad8d7d41a7e45913. Latest consensus state: current_term: 1 leader_uuid: "233e61279a444325ad8d7d41a7e45913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e61279a444325ad8d7d41a7e45913" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 40319 } } }
I20260926 02:20:13.254737 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:13.254557 17997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "233e61279a444325ad8d7d41a7e45913" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "233e61279a444325ad8d7d41a7e45913" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 40319 } } }
I20260926 02:20:13.254915 17997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:13.259584 18005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:13.265151 18005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:13.275511 18005 catalog_manager.cc:1347] Generated new cluster ID: 7c44a95d3d56437bb40bf1ab5e817026
I20260926 02:20:13.275650 18005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:13.291643 18005 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:13.292106 18005 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:13.309160 18005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913: Generated new TSK 0
I20260926 02:20:13.310112 18005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:20:13.320587 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:40319
--builtin_ntp_servers=127.12.244.212:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:20:13.675230 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:13.675557 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:13.675709 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:13.683652 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:13.683771 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:13.683961 18015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:13.693979 18015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:40319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:13.695518 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:13.697410 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:13.711772 18020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:13.711673 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:13.711721 18021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:13.712976 18015 server_base.cc:1034] running on GCE node
I20260926 02:20:13.713697 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:13.714803 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:13.716104 18015 hybrid_clock.cc:648] HybridClock initialized: now 1790389213716012 us; error 93 us; skew 500 ppm
I20260926 02:20:13.716482 18015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:13.719027 18015 webserver.cc:460] Webserver started at http://127.12.244.193:33169/ using document root <none> and password file <none>
I20260926 02:20:13.719857 18015 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:13.720006 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:13.720355 18015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:13.723632 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "c945822a50c644618d1920aa643a8b6a"
format_stamp: "Formatted at 2026-09-26 02:20:13 on dist-test-slave-qfmw"
server_key: "668852f32dd92d818ecdf40b0dbe2fd4"
server_key_iv: "096eb6989212bf91946f5bafa501f855"
server_key_version: "encryptionkey@0"
I20260926 02:20:13.724509 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "c945822a50c644618d1920aa643a8b6a"
format_stamp: "Formatted at 2026-09-26 02:20:13 on dist-test-slave-qfmw"
server_key: "668852f32dd92d818ecdf40b0dbe2fd4"
server_key_iv: "096eb6989212bf91946f5bafa501f855"
server_key_version: "encryptionkey@0"
I20260926 02:20:13.730206 18015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:20:13.734440 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:13.736109 18015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:20:13.736325 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "c945822a50c644618d1920aa643a8b6a"
format_stamp: "Formatted at 2026-09-26 02:20:13 on dist-test-slave-qfmw"
server_key: "668852f32dd92d818ecdf40b0dbe2fd4"
server_key_iv: "096eb6989212bf91946f5bafa501f855"
server_key_version: "encryptionkey@0"
I20260926 02:20:13.736608 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:13.743001 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:13.743932 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:13.744179 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:13.745159 18015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:13.746625 18015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:13.746712 18015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:13.746788 18015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:13.746842 18015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:13.781714 18015 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:32815
I20260926 02:20:13.781778 18141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:32815 every 8 connection(s)
I20260926 02:20:13.783160 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:13.784437 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18015
I20260926 02:20:13.784649 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:13.785393 13267 external_mini_cluster.cc:1428] Setting key 4ca278d907f307aba4e7de21279405fe
I20260926 02:20:13.788517 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:40319
--builtin_ntp_servers=127.12.244.212:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:20:13.792829 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:13.786134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:57039 (local address 127.12.244.254:40319)
0926 02:20:13.786655 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:13.786660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:13.788451 (+ 1791us) server_negotiation.cc:407] Connection header received
0926 02:20:13.790250 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:13.790258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:13.790341 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:13.790465 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:13.791928 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:13.791936 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:13.791945 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:13.791971 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:13.792028 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:13.792049 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:13.792178 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:13.792353 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 02:20:13.792423 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":289,"threads_started":1}
I20260926 02:20:13.793808 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:13.787030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:40319 (local address 127.12.244.193:57039)
0926 02:20:13.788253 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:13.788302 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:13.789935 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:13.790701 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:13.790710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:13.790820 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:13.791590 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:13.791593 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:13.791635 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:13.791730 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:13.792149 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:13.792156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:13.792162 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:20:13.792413 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 02:20:13.792655 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":372,"threads_started":1}
I20260926 02:20:13.795619 18142 heartbeater.cc:347] Connected to a master server at 127.12.244.254:40319
I20260926 02:20:13.795984 18142 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:13.796710 18142 heartbeater.cc:511] Master 127.12.244.254:40319 requested a full tablet report, sending...
I20260926 02:20:13.799172 17956 ts_manager.cc:194] Registered new tserver with Master: c945822a50c644618d1920aa643a8b6a (127.12.244.193:32815)
W20260926 02:20:14.131206 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:14.131565 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:14.131729 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:14.139451 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:14.139595 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:14.139778 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:14.149322 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:40319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:14.150869 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:14.153190 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:14.167821 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:14.167769 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:14.167737 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:14.168740 18146 server_base.cc:1034] running on GCE node
I20260926 02:20:14.169432 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:14.170579 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:14.171903 18146 hybrid_clock.cc:648] HybridClock initialized: now 1790389214171828 us; error 74 us; skew 500 ppm
I20260926 02:20:14.172267 18146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:14.174817 18146 webserver.cc:460] Webserver started at http://127.12.244.194:46795/ using document root <none> and password file <none>
I20260926 02:20:14.175653 18146 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:14.175804 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:14.176158 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:14.179941 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "33299694328843b2b17dbe83be197dbf"
format_stamp: "Formatted at 2026-09-26 02:20:14 on dist-test-slave-qfmw"
server_key: "3a5bf50f07d76cc8e4656d1f651a309f"
server_key_iv: "4d63918be1f661878df9508640ed77d6"
server_key_version: "encryptionkey@0"
I20260926 02:20:14.180891 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "33299694328843b2b17dbe83be197dbf"
format_stamp: "Formatted at 2026-09-26 02:20:14 on dist-test-slave-qfmw"
server_key: "3a5bf50f07d76cc8e4656d1f651a309f"
server_key_iv: "4d63918be1f661878df9508640ed77d6"
server_key_version: "encryptionkey@0"
I20260926 02:20:14.186854 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:20:14.191411 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:14.193107 18146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:20:14.193346 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "33299694328843b2b17dbe83be197dbf"
format_stamp: "Formatted at 2026-09-26 02:20:14 on dist-test-slave-qfmw"
server_key: "3a5bf50f07d76cc8e4656d1f651a309f"
server_key_iv: "4d63918be1f661878df9508640ed77d6"
server_key_version: "encryptionkey@0"
I20260926 02:20:14.193621 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:14.200147 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:14.201164 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:14.201468 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:14.202524 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:14.204313 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:14.204443 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:14.204588 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:14.204679 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:14.241637 18146 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:38437
I20260926 02:20:14.241672 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:38437 every 8 connection(s)
I20260926 02:20:14.243265 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:14.251931 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18146
I20260926 02:20:14.252138 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:14.252744 13267 external_mini_cluster.cc:1428] Setting key 1071df252dfd46e2ce4f47354f301ab5
I20260926 02:20:14.252772 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.246245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:38915 (local address 127.12.244.254:40319)
0926 02:20:14.246447 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.246452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.248068 (+ 1616us) server_negotiation.cc:407] Connection header received
0926 02:20:14.249889 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.249893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.249948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.250040 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.251700 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.251760 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.251765 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.251792 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.251836 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.251864 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.252018 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.252366 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.252454 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:20:14.254011 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.246817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:40319 (local address 127.12.244.194:38915)
0926 02:20:14.247857 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:14.247937 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:14.249603 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:14.250277 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:14.250286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:14.250394 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:14.251258 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:14.251261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:14.251309 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:14.251421 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:14.252020 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:14.252034 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:14.252047 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 02:20:14.252399 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 02:20:14.252701 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":339,"threads_started":1}
I20260926 02:20:14.255681 18273 heartbeater.cc:347] Connected to a master server at 127.12.244.254:40319
I20260926 02:20:14.256040 18273 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:14.256084 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:40319
--builtin_ntp_servers=127.12.244.212:35459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:20:14.256875 18273 heartbeater.cc:511] Master 127.12.244.254:40319 requested a full tablet report, sending...
I20260926 02:20:14.258615 17956 ts_manager.cc:194] Registered new tserver with Master: 33299694328843b2b17dbe83be197dbf (127.12.244.194:38437)
W20260926 02:20:14.606138 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:14.606467 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:14.606560 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:14.614140 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:14.614274 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:14.614387 18276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:14.623858 18276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:40319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:14.625195 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:14.627230 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:14.641907 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:14.641939 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:14.641901 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:14.642822 18276 server_base.cc:1034] running on GCE node
I20260926 02:20:14.643559 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:14.644604 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:14.645829 18276 hybrid_clock.cc:648] HybridClock initialized: now 1790389214645767 us; error 45 us; skew 500 ppm
I20260926 02:20:14.646198 18276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:14.648767 18276 webserver.cc:460] Webserver started at http://127.12.244.195:44215/ using document root <none> and password file <none>
I20260926 02:20:14.649551 18276 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:14.649672 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:14.650048 18276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:14.653311 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "8a23e7a2dbf94ec691bb089d1cdca536"
format_stamp: "Formatted at 2026-09-26 02:20:14 on dist-test-slave-qfmw"
server_key: "e05eab5b9d07d755b194b8b65bccdf4e"
server_key_iv: "002f6bb221727d2e70b6b97ff5a65ffd"
server_key_version: "encryptionkey@0"
I20260926 02:20:14.654176 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "8a23e7a2dbf94ec691bb089d1cdca536"
format_stamp: "Formatted at 2026-09-26 02:20:14 on dist-test-slave-qfmw"
server_key: "e05eab5b9d07d755b194b8b65bccdf4e"
server_key_iv: "002f6bb221727d2e70b6b97ff5a65ffd"
server_key_version: "encryptionkey@0"
I20260926 02:20:14.660012 18276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:20:14.664072 18290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:14.665691 18276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:14.665916 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "8a23e7a2dbf94ec691bb089d1cdca536"
format_stamp: "Formatted at 2026-09-26 02:20:14 on dist-test-slave-qfmw"
server_key: "e05eab5b9d07d755b194b8b65bccdf4e"
server_key_iv: "002f6bb221727d2e70b6b97ff5a65ffd"
server_key_version: "encryptionkey@0"
I20260926 02:20:14.666174 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:14.672554 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:14.673472 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:14.673768 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:14.674777 18276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:14.676394 18276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:14.676482 18276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:14.676647 18276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:14.676738 18276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:14.717499 18276 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:40501
I20260926 02:20:14.717533 18402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:40501 every 8 connection(s)
I20260926 02:20:14.718940 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:14.727674 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18276
I20260926 02:20:14.727854 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:14.727936 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.721947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:53879 (local address 127.12.244.254:40319)
0926 02:20:14.722121 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.722126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.723663 (+ 1537us) server_negotiation.cc:407] Connection header received
0926 02:20:14.725470 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.725475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.725557 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.725674 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.727151 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.727156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.727161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.727181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.727220 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.727235 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.727392 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.727579 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.727660 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:20:14.728379 13267 external_mini_cluster.cc:1428] Setting key ca748171b72dfd7f9bbe929c71e6f564
I20260926 02:20:14.729130 18405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.722424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:40319 (local address 127.12.244.195:53879)
0926 02:20:14.723488 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:14.723527 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:14.725154 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:14.725939 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:14.725949 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:14.726060 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:14.726775 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:14.726777 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:14.726851 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:14.726945 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:14.727335 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:14.727346 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:14.727356 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:20:14.727655 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 02:20:14.727947 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":313,"threads_started":1}
I20260926 02:20:14.730698 18403 heartbeater.cc:347] Connected to a master server at 127.12.244.254:40319
I20260926 02:20:14.731069 18403 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:14.731791 18403 heartbeater.cc:511] Master 127.12.244.254:40319 requested a full tablet report, sending...
I20260926 02:20:14.733258 17956 ts_manager.cc:194] Registered new tserver with Master: 8a23e7a2dbf94ec691bb089d1cdca536 (127.12.244.195:40501)
I20260926 02:20:14.744233 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:20:14.751922 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.749189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.12.244.254:40319)
0926 02:20:14.749422 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.749431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.749843 (+ 412us) server_negotiation.cc:407] Connection header received
0926 02:20:14.749982 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.749985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.750038 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.750120 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.751168 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.751172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.751175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.751191 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.751223 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.751237 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.751351 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.751523 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.751577 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:20:14.756163 17956 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:20:14.763242 17956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:14.782871 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.778172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.12.244.193:32815)
0926 02:20:14.779040 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.779059 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.779119 (+ 60us) server_negotiation.cc:407] Connection header received
0926 02:20:14.779193 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.779201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.779442 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.780412 (+ 970us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.781427 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.781434 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.781446 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.781463 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.781528 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.781550 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.781746 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.782370 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.782485 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":602,"threads_started":1}
I20260926 02:20:14.783207 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.777700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:52972)
0926 02:20:14.778402 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:14.778425 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:14.778548 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:14.779613 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:14.779622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:14.779717 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:14.781011 (+ 1294us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:14.781015 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:14.781031 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:14.781099 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:14.781694 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:14.781701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:14.781706 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:20:14.781888 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 02:20:14.781946 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":354,"threads_started":1}
I20260926 02:20:14.784541 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.776955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:38437 (local address 127.0.0.1:49452)
0926 02:20:14.777791 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:14.777840 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:14.778008 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:14.779613 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:14.779622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:14.779726 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:14.780856 (+ 1130us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:14.780863 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:14.780899 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:14.781007 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:14.781575 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:14.781585 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:14.781593 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:20:14.781775 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 02:20:14.781878 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":341,"threads_started":1}
I20260926 02:20:14.785424 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.778433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:40501 (local address 127.0.0.1:35456)
0926 02:20:14.779327 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:14.779342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:14.779426 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:14.781897 (+ 2471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:14.781902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:14.781932 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:14.782841 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:14.782845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:14.782861 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:14.782902 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:14.784764 (+ 1862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:14.784770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:14.784783 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 02:20:14.784927 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 02:20:14.784987 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":691,"threads_started":1}
I20260926 02:20:14.785449 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.777205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49452 (local address 127.12.244.194:38437)
0926 02:20:14.777778 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.777787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.778159 (+ 372us) server_negotiation.cc:407] Connection header received
0926 02:20:14.778270 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.778282 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.778562 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.778735 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.781311 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.781321 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.781336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.781367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.781429 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.781452 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.781608 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.785023 (+ 3415us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.785139 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":665,"threads_started":1}
I20260926 02:20:14.787766 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.778937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.12.244.195:40501)
0926 02:20:14.780236 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.780253 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.780287 (+ 34us) server_negotiation.cc:407] Connection header received
0926 02:20:14.780351 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.780362 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.780578 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.783404 (+ 2826us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.784484 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.784494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.784509 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.784527 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.784590 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.784610 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.787149 (+ 2539us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.787232 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.787343 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":962,"threads_started":1}
I20260926 02:20:14.788653 18077 tablet_service.cc:1467] Processing CreateTablet for tablet d0b9ae2128da4a5981c8635fa64e978b (DEFAULT_TABLE table=test-table [id=00e6ed8bea0843b983fe1308f1a03fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:14.789580 18208 tablet_service.cc:1467] Processing CreateTablet for tablet d0b9ae2128da4a5981c8635fa64e978b (DEFAULT_TABLE table=test-table [id=00e6ed8bea0843b983fe1308f1a03fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:14.790934 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b9ae2128da4a5981c8635fa64e978b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:14.791076 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b9ae2128da4a5981c8635fa64e978b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:14.791917 18338 tablet_service.cc:1467] Processing CreateTablet for tablet d0b9ae2128da4a5981c8635fa64e978b (DEFAULT_TABLE table=test-table [id=00e6ed8bea0843b983fe1308f1a03fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:14.793532 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b9ae2128da4a5981c8635fa64e978b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:14.802778 18142 heartbeater.cc:503] Master 127.12.244.254:40319 was elected leader, sending a full tablet report...
I20260926 02:20:14.803166 18421 tablet_bootstrap.cc:492] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Bootstrap starting.
I20260926 02:20:14.806703 18421 tablet_bootstrap.cc:654] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:14.807910 18421 log.cc:824] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:14.809906 18422 tablet_bootstrap.cc:492] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: Bootstrap starting.
I20260926 02:20:14.810995 18421 tablet_bootstrap.cc:492] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: No bootstrap required, opened a new log
I20260926 02:20:14.811312 18421 ts_tablet_manager.cc:1397] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260926 02:20:14.814283 18422 tablet_bootstrap.cc:654] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:14.816236 18422 log.cc:824] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:14.817334 18424 tablet_bootstrap.cc:492] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Bootstrap starting.
I20260926 02:20:14.820606 18421 raft_consensus.cc:348] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.821101 18421 raft_consensus.cc:374] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:14.821241 18421 raft_consensus.cc:729] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33299694328843b2b17dbe83be197dbf, State: Initialized, Role: FOLLOWER
I20260926 02:20:14.821491 18424 tablet_bootstrap.cc:654] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:14.822037 18422 tablet_bootstrap.cc:492] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: No bootstrap required, opened a new log
I20260926 02:20:14.822072 18421 consensus_queue.cc:260] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.822345 18422 ts_tablet_manager.cc:1397] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 02:20:14.822907 18424 log.cc:824] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:14.824640 18273 heartbeater.cc:503] Master 127.12.244.254:40319 was elected leader, sending a full tablet report...
I20260926 02:20:14.826671 18424 tablet_bootstrap.cc:492] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: No bootstrap required, opened a new log
I20260926 02:20:14.826915 18421 ts_tablet_manager.cc:1428] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260926 02:20:14.826985 18424 ts_tablet_manager.cc:1397] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260926 02:20:14.833019 18422 raft_consensus.cc:348] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.833585 18422 raft_consensus.cc:374] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:14.833763 18422 raft_consensus.cc:729] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c945822a50c644618d1920aa643a8b6a, State: Initialized, Role: FOLLOWER
I20260926 02:20:14.834573 18422 consensus_queue.cc:260] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.835966 18424 raft_consensus.cc:348] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.836489 18424 raft_consensus.cc:374] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:14.836611 18424 raft_consensus.cc:729] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a23e7a2dbf94ec691bb089d1cdca536, State: Initialized, Role: FOLLOWER
I20260926 02:20:14.836825 18422 ts_tablet_manager.cc:1428] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260926 02:20:14.837289 18424 consensus_queue.cc:260] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.838518 18403 heartbeater.cc:503] Master 127.12.244.254:40319 was elected leader, sending a full tablet report...
I20260926 02:20:14.839244 18424 ts_tablet_manager.cc:1428] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260926 02:20:14.845757 18426 raft_consensus.cc:482] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:14.846141 18426 raft_consensus.cc:504] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.848078 18426 leader_election.cc:288] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c945822a50c644618d1920aa643a8b6a (127.12.244.193:32815), 8a23e7a2dbf94ec691bb089d1cdca536 (127.12.244.195:40501)
I20260926 02:20:14.851449 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.848470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:57165 (local address 127.12.244.193:32815)
0926 02:20:14.848637 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.848641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.849173 (+ 532us) server_negotiation.cc:407] Connection header received
0926 02:20:14.849469 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.849474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.849545 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.849605 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.850773 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.850779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.850784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.850803 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.850845 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.850856 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.850980 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.851055 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.851147 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:20:14.851547 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.849431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:45189 (local address 127.12.244.195:40501)
0926 02:20:14.849613 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.849620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.849750 (+ 130us) server_negotiation.cc:407] Connection header received
0926 02:20:14.849953 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.849958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.850012 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.850113 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.850873 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.850879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.850883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.850900 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.850932 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.850953 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.851078 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.851185 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.851259 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:20:14.851647 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.848281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.12.244.194:57165)
0926 02:20:14.849065 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:14.849084 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:14.849291 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:14.849760 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:14.849765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:14.849783 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:14.850540 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:14.850545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:14.850566 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:14.850616 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:14.850950 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:14.850955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:14.850959 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:20:14.851109 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 02:20:14.851192 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":367,"threads_started":1}
I20260926 02:20:14.851742 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.849058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:40501 (local address 127.12.244.194:45189)
0926 02:20:14.849635 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:14.849658 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:14.849794 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:14.850130 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:14.850134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:14.850158 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:14.850697 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:20:14.850699 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:14.850710 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:20:14.850743 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:14.851044 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:14.851048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:14.851053 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:20:14.851221 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 02:20:14.851269 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":286,"threads_started":1}
I20260926 02:20:14.852703 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9ae2128da4a5981c8635fa64e978b" candidate_uuid: "33299694328843b2b17dbe83be197dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c945822a50c644618d1920aa643a8b6a" is_pre_election: true
I20260926 02:20:14.852702 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9ae2128da4a5981c8635fa64e978b" candidate_uuid: "33299694328843b2b17dbe83be197dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" is_pre_election: true
I20260926 02:20:14.853227 18358 raft_consensus.cc:2454] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33299694328843b2b17dbe83be197dbf in term 0.
I20260926 02:20:14.853227 18097 raft_consensus.cc:2454] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33299694328843b2b17dbe83be197dbf in term 0.
I20260926 02:20:14.853894 18164 leader_election.cc:302] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33299694328843b2b17dbe83be197dbf, c945822a50c644618d1920aa643a8b6a; no voters:
I20260926 02:20:14.854280 18426 raft_consensus.cc:2781] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:14.854455 18426 raft_consensus.cc:482] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:14.854588 18426 raft_consensus.cc:3037] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:14.857044 18426 raft_consensus.cc:504] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.857677 18426 leader_election.cc:288] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [CANDIDATE]: Term 1 election: Requested vote from peers c945822a50c644618d1920aa643a8b6a (127.12.244.193:32815), 8a23e7a2dbf94ec691bb089d1cdca536 (127.12.244.195:40501)
I20260926 02:20:14.858464 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9ae2128da4a5981c8635fa64e978b" candidate_uuid: "33299694328843b2b17dbe83be197dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c945822a50c644618d1920aa643a8b6a"
I20260926 02:20:14.858510 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b9ae2128da4a5981c8635fa64e978b" candidate_uuid: "33299694328843b2b17dbe83be197dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a23e7a2dbf94ec691bb089d1cdca536"
I20260926 02:20:14.858698 18097 raft_consensus.cc:3037] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:14.858724 18358 raft_consensus.cc:3037] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:14.861100 18358 raft_consensus.cc:2454] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33299694328843b2b17dbe83be197dbf in term 1.
I20260926 02:20:14.861099 18097 raft_consensus.cc:2454] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33299694328843b2b17dbe83be197dbf in term 1.
I20260926 02:20:14.861624 18164 leader_election.cc:302] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33299694328843b2b17dbe83be197dbf, c945822a50c644618d1920aa643a8b6a; no voters:
I20260926 02:20:14.861996 18426 raft_consensus.cc:2781] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:14.862560 18426 raft_consensus.cc:686] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 1 LEADER]: Becoming Leader. State: Replica: 33299694328843b2b17dbe83be197dbf, State: Running, Role: LEADER
I20260926 02:20:14.863194 18426 consensus_queue.cc:237] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } }
I20260926 02:20:14.868300 17956 catalog_manager.cc:5495] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 33299694328843b2b17dbe83be197dbf (127.12.244.194). New cstate: current_term: 1 leader_uuid: "33299694328843b2b17dbe83be197dbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33299694328843b2b17dbe83be197dbf" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 38437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 } health_report { overall_health: UNKNOWN } } }
I20260926 02:20:14.892813 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.890609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49454 (local address 127.12.244.194:38437)
0926 02:20:14.890850 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.890854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.890873 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:20:14.891036 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.891040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.891146 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.891235 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.892168 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.892173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.892176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.892191 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.892223 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.892235 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.892339 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.892429 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.892487 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:20:14.902518 18358 raft_consensus.cc:1261] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 1 FOLLOWER]: Refusing update from remote peer 33299694328843b2b17dbe83be197dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:14.902518 18097 raft_consensus.cc:1261] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 1 FOLLOWER]: Refusing update from remote peer 33299694328843b2b17dbe83be197dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:14.903697 18432 consensus_queue.cc:1035] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a23e7a2dbf94ec691bb089d1cdca536" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:14.904157 18426 consensus_queue.cc:1035] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c945822a50c644618d1920aa643a8b6a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:14.915503 18435 mvcc.cc:205] Tried to move back new op lower bound from 7333434224229351424 to 7333434224084492288. Current Snapshot: MvccSnapshot[applied={T|T < 7333434224229351424}]
I20260926 02:20:14.916316 18437 mvcc.cc:205] Tried to move back new op lower bound from 7333434224229351424 to 7333434224084492288. Current Snapshot: MvccSnapshot[applied={T|T < 7333434224229351424}]
I20260926 02:20:14.935950 17956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:14.936233 17956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:14.940366 17956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 233e61279a444325ad8d7d41a7e45913: Sending DeleteTablet for 3 replicas of tablet d0b9ae2128da4a5981c8635fa64e978b
I20260926 02:20:14.941861 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b9ae2128da4a5981c8635fa64e978b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:14 UTC) from {username='slave'} at 127.0.0.1:49452
I20260926 02:20:14.941923 18077 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b9ae2128da4a5981c8635fa64e978b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:14 UTC) from {username='slave'} at 127.0.0.1:52972
I20260926 02:20:14.942911 18338 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b9ae2128da4a5981c8635fa64e978b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:14 UTC) from {username='slave'} at 127.0.0.1:35456
I20260926 02:20:14.943488 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18015
I20260926 02:20:14.951714 18447 tablet_replica.cc:330] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: stopping tablet replica
I20260926 02:20:14.952527 18447 raft_consensus.cc:2229] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:14.953378 18447 raft_consensus.cc:2258] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:14.958451 18445 tablet_replica.cc:330] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: stopping tablet replica
I20260926 02:20:14.958622 18446 tablet_replica.cc:330] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: stopping tablet replica
I20260926 02:20:14.958981 18446 raft_consensus.cc:2229] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:14.958988 18445 raft_consensus.cc:2229] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:14.959560 18446 raft_consensus.cc:2258] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:14.959772 18445 raft_consensus.cc:2258] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:14.961397 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:14.945429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.12.244.193:32815)
0926 02:20:14.945663 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:14.945672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:14.956127 (+ 10455us) server_negotiation.cc:407] Connection header received
0926 02:20:14.956816 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:14.956826 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:14.956990 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:14.957088 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:14.960496 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:14.960502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:14.960507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:14.960538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:14.960591 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:14.960617 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:14.960751 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:14.961026 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 02:20:14.961095 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:20:14.962188 18446 ts_tablet_manager.cc:1905] T d0b9ae2128da4a5981c8635fa64e978b P c945822a50c644618d1920aa643a8b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:14.963124 18137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:14.963762 18447 ts_tablet_manager.cc:1905] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:14.967917 18445 ts_tablet_manager.cc:1905] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:14.973462 18447 ts_tablet_manager.cc:1918] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:14.973762 18447 log.cc:1196] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/d0b9ae2128da4a5981c8635fa64e978b
I20260926 02:20:14.974412 18447 ts_tablet_manager.cc:1939] T d0b9ae2128da4a5981c8635fa64e978b P 8a23e7a2dbf94ec691bb089d1cdca536: Deleting consensus metadata
I20260926 02:20:14.976429 18445 ts_tablet_manager.cc:1918] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:14.976749 18445 log.cc:1196] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/d0b9ae2128da4a5981c8635fa64e978b
I20260926 02:20:14.976781 17943 catalog_manager.cc:4841] TS 8a23e7a2dbf94ec691bb089d1cdca536 (127.12.244.195:40501): tablet d0b9ae2128da4a5981c8635fa64e978b (table test-table [id=00e6ed8bea0843b983fe1308f1a03fa0]) successfully deleted
I20260926 02:20:14.977481 18445 ts_tablet_manager.cc:1939] T d0b9ae2128da4a5981c8635fa64e978b P 33299694328843b2b17dbe83be197dbf: Deleting consensus metadata
I20260926 02:20:14.979454 17942 catalog_manager.cc:4841] TS 33299694328843b2b17dbe83be197dbf (127.12.244.194:38437): tablet d0b9ae2128da4a5981c8635fa64e978b (table test-table [id=00e6ed8bea0843b983fe1308f1a03fa0]) successfully deleted
I20260926 02:20:15.064299 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18015
W20260926 02:20:15.081939 17943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.244.193:32815 (error 108)
I20260926 02:20:15.083393 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18146
I20260926 02:20:15.083931 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.083422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:52986)
0926 02:20:15.083599 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:15.083734 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:20:15.084559 17943 catalog_manager.cc:4584] TS c945822a50c644618d1920aa643a8b6a (127.12.244.193:32815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0b9ae2128da4a5981c8635fa64e978b: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
I20260926 02:20:15.086790 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.084310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.12.244.194:38437)
0926 02:20:15.084512 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:15.084518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:15.084541 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:20:15.084663 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:15.084667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:15.084713 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:15.084843 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:15.085974 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:15.085980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:15.085985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:15.086006 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:15.086045 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:15.086061 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:15.086220 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:15.086403 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 02:20:15.086473 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:20:15.087894 18268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:15.138551 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.138027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:52990)
0926 02:20:15.138265 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:15.138369 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 02:20:15.139791 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.139396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:53002)
0926 02:20:15.139560 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:15.139661 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:20:15.199400 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18146
I20260926 02:20:15.201594 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.201032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:53012)
0926 02:20:15.201230 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:15.201381 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 02:20:15.202677 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.202313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:53026)
0926 02:20:15.202476 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:15.202559 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:20:15.217875 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18276
I20260926 02:20:15.220960 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.218722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.12.244.195:40501)
0926 02:20:15.219068 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:15.219072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:15.219086 (+ 14us) server_negotiation.cc:407] Connection header received
0926 02:20:15.219196 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:15.219200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:15.219271 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:15.219380 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:15.220315 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:15.220319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:15.220322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:15.220343 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:15.220375 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:15.220386 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:15.220528 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:15.220626 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 02:20:15.220685 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260926 02:20:15.221730 18398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:15.283322 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.282687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:53034)
0926 02:20:15.282957 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:15.283077 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 02:20:15.284638 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.284153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:32815 (local address 127.0.0.1:53042)
0926 02:20:15.284399 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:15.284489 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 02:20:15.312125 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18276
I20260926 02:20:15.328554 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 17925
I20260926 02:20:15.329743 17986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:15.436527 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 17925
2026-09-26T02:20:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2738 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 02:20:15.480298 13267 mini_oidc.cc:152] Starting JWKS server
I20260926 02:20:15.483363 13267 webserver.cc:460] Webserver started at https://127.0.0.1:40951/ using document root <none> and password file <none>
I20260926 02:20:15.483722 13267 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:20:15.484432 13267 webserver.cc:460] Webserver started at http://127.0.0.1:36329/ using document root <none> and password file <none>
2026-09-26T02:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:15Z Disabled control of system clock
I20260926 02:20:15.514542 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39021
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39021
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40951/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/testchainca.cert with env {}
W20260926 02:20:15.878877 18459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:15.879329 18459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:15.879493 18459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:15.887481 18459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:15.887640 18459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:15.887732 18459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:15.887850 18459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:15.887928 18459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:20:15.888016 18459 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:20:15.888091 18459 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40951/jwks.json
I20260926 02:20:15.898515 18459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40951/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:15.900049 18459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:15.901449 18459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:15.914105 18464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:15.914134 18465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:15.914477 18467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:15.915241 18459 server_base.cc:1034] running on GCE node
I20260926 02:20:15.916131 18459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:15.917629 18459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:15.918877 18459 hybrid_clock.cc:648] HybridClock initialized: now 1790389215918812 us; error 54 us; skew 500 ppm
I20260926 02:20:15.919324 18459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:15.921741 18459 webserver.cc:460] Webserver started at http://127.12.244.254:46505/ using document root <none> and password file <none>
I20260926 02:20:15.922465 18459 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:15.922564 18459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:15.922832 18459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:15.926043 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3"
format_stamp: "Formatted at 2026-09-26 02:20:15 on dist-test-slave-qfmw"
server_key: "3ac10d35fd3d18dff1d0c5f0c844beff"
server_key_iv: "217ff057b32ffde82cfeaca7002760a2"
server_key_version: "encryptionkey@0"
I20260926 02:20:15.926900 18459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3"
format_stamp: "Formatted at 2026-09-26 02:20:15 on dist-test-slave-qfmw"
server_key: "3ac10d35fd3d18dff1d0c5f0c844beff"
server_key_iv: "217ff057b32ffde82cfeaca7002760a2"
server_key_version: "encryptionkey@0"
I20260926 02:20:15.932662 18459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:20:15.936491 18473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:15.938103 18459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:15.938313 18459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3"
format_stamp: "Formatted at 2026-09-26 02:20:15 on dist-test-slave-qfmw"
server_key: "3ac10d35fd3d18dff1d0c5f0c844beff"
server_key_iv: "217ff057b32ffde82cfeaca7002760a2"
server_key_version: "encryptionkey@0"
I20260926 02:20:15.938568 18459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:15.956849 18459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:15.957793 18459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:15.958135 18459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:15.977604 18459 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:39021
I20260926 02:20:15.977623 18524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:39021 every 8 connection(s)
I20260926 02:20:15.979152 18459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:15.983276 18525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:15.986644 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18459
I20260926 02:20:15.986909 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:15.987610 13267 external_mini_cluster.cc:1428] Setting key 10eb271fd71732f5dbfaefdae26e94d5
I20260926 02:20:15.997802 18525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3: Bootstrap starting.
I20260926 02:20:16.001896 18525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:16.003417 18525 log.cc:824] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:16.004264 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:15.989802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.12.244.254:39021)
0926 02:20:15.990920 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:15.990936 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:15.990991 (+ 55us) server_negotiation.cc:407] Connection header received
0926 02:20:15.992936 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:15.992977 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:15.993713 (+ 736us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:15.994284 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:15.994925 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:15.996962 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:15.998432 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:15.999509 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:15.999737 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:16.000929 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:16.000942 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:16.000962 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:16.001692 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:16.001869 (+ 177us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:16.001905 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:16.002042 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:16.002382 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 02:20:16.002783 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":356,"threads_started":1}
I20260926 02:20:16.006844 18525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3: No bootstrap required, opened a new log
I20260926 02:20:16.008600 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:20:16.015064 18525 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39021 } }
I20260926 02:20:16.015612 18525 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:16.015722 18525 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5efd3b5e7ad4f9f9d4d8c1eeadc10d3, State: Initialized, Role: FOLLOWER
I20260926 02:20:16.016451 18525 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39021 } }
I20260926 02:20:16.016733 18525 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:16.016875 18525 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:16.017048 18525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:16.019666 18525 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39021 } }
I20260926 02:20:16.020340 18525 leader_election.cc:302] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5efd3b5e7ad4f9f9d4d8c1eeadc10d3; no voters:
I20260926 02:20:16.020996 18525 leader_election.cc:288] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:16.021229 18530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:16.022625 18530 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [term 1 LEADER]: Becoming Leader. State: Replica: d5efd3b5e7ad4f9f9d4d8c1eeadc10d3, State: Running, Role: LEADER
I20260926 02:20:16.023306 18530 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39021 } }
I20260926 02:20:16.023820 18525 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:16.026335 18531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39021 } } }
I20260926 02:20:16.026810 18531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:16.026414 18532 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5efd3b5e7ad4f9f9d4d8c1eeadc10d3. Latest consensus state: current_term: 1 leader_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5efd3b5e7ad4f9f9d4d8c1eeadc10d3" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39021 } } }
I20260926 02:20:16.027323 18532 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:16.030157 18539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:16.034916 18539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:16.051514 18539 catalog_manager.cc:1347] Generated new cluster ID: fbd3b991a9cf495db8dd14d0dc9c485f
I20260926 02:20:16.051651 18539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:16.067272 18539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:16.069272 18539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:16.081097 18539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5efd3b5e7ad4f9f9d4d8c1eeadc10d3: Generated new TSK 0
I20260926 02:20:16.082050 18539 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:20:16.135054 18555 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 02:20:16.135341 18555 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:20:16.136363 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:16.107357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.12.244.254:39021)
0926 02:20:16.107631 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:16.107646 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:16.108268 (+ 622us) server_negotiation.cc:407] Connection header received
0926 02:20:16.108633 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:16.108637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:16.108721 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:16.108883 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:20:16.109441 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.110182 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.111730 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.112200 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.112370 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:16.112963 (+ 593us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:20:16.135600 (+ 22637us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:20:16.136020 (+ 420us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43640: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 02:20:16.136504 18528 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43640: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 02:20:16.137979 18554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:16.107326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39021 (local address 127.0.0.1:43640)
0926 02:20:16.108153 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:16.108170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:16.108380 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:16.108920 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:16.108926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:16.108940 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:20:16.109286 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:16.109294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.110334 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.110340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:16.111581 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:16.111593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.112447 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.112457 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:16.112598 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:16.112661 (+ 63us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:20:16.135926 (+ 23265us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:20:16.136301 (+ 375us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.244.254:39021: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":492,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.244.254:39021: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40951/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 02:20:16.140015 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18459
I20260926 02:20:16.141427 18520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:16.236631 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18459
2026-09-26T02:20:16Z chronyd exiting
I20260926 02:20:16.287844 13267 test_util.cc:175] -----------------------------------------------
I20260926 02:20:16.288005 13267 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790389169147784-13267-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (812 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 02:20:16.291803 13267 mini_oidc.cc:152] Starting JWKS server
I20260926 02:20:16.293663 13267 webserver.cc:460] Webserver started at https://127.0.0.1:35893/ using document root <none> and password file <none>
I20260926 02:20:16.293922 13267 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:20:16.294621 13267 webserver.cc:460] Webserver started at http://127.0.0.1:44037/ using document root <none> and password file <none>
2026-09-26T02:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:16Z Disabled control of system clock
I20260926 02:20:16.322611 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:44931
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:44931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35893/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/testchainca.cert with env {}
W20260926 02:20:16.661577 18562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:16.661964 18562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:16.662117 18562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:16.669476 18562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:16.669658 18562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:16.669761 18562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:16.669875 18562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:16.669970 18562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:20:16.670058 18562 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:20:16.670131 18562 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35893/jwks.json
I20260926 02:20:16.680590 18562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:44931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:44931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35893/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:16.681988 18562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:16.683378 18562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:16.696238 18567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:16.696303 18568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:16.696794 18570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:16.701170 18562 server_base.cc:1034] running on GCE node
I20260926 02:20:16.702396 18562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:16.704620 18562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:16.705905 18562 hybrid_clock.cc:648] HybridClock initialized: now 1790389216705827 us; error 59 us; skew 500 ppm
I20260926 02:20:16.706404 18562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:16.709103 18562 webserver.cc:460] Webserver started at http://127.12.244.254:43039/ using document root <none> and password file <none>
I20260926 02:20:16.709896 18562 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:16.710032 18562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:16.710399 18562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:16.713603 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "758eba72959e4f709e66da5705401f4b"
format_stamp: "Formatted at 2026-09-26 02:20:16 on dist-test-slave-qfmw"
server_key: "78cad26407b788d6dfe97494f7da35e8"
server_key_iv: "c2e6bdebf7912dc951860f63067cc83b"
server_key_version: "encryptionkey@0"
I20260926 02:20:16.714457 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "758eba72959e4f709e66da5705401f4b"
format_stamp: "Formatted at 2026-09-26 02:20:16 on dist-test-slave-qfmw"
server_key: "78cad26407b788d6dfe97494f7da35e8"
server_key_iv: "c2e6bdebf7912dc951860f63067cc83b"
server_key_version: "encryptionkey@0"
I20260926 02:20:16.720135 18562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:20:16.723969 18576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:16.725531 18562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:16.725745 18562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "758eba72959e4f709e66da5705401f4b"
format_stamp: "Formatted at 2026-09-26 02:20:16 on dist-test-slave-qfmw"
server_key: "78cad26407b788d6dfe97494f7da35e8"
server_key_iv: "c2e6bdebf7912dc951860f63067cc83b"
server_key_version: "encryptionkey@0"
I20260926 02:20:16.725988 18562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:16.744213 18562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:16.745105 18562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:16.745427 18562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:16.764396 18562 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:44931
I20260926 02:20:16.764420 18627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:44931 every 8 connection(s)
I20260926 02:20:16.765918 18562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:16.769852 18628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:16.773540 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18562
I20260926 02:20:16.773751 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:16.774550 13267 external_mini_cluster.cc:1428] Setting key 52e0f84e2d9da2fcf5c35ebeddf01fc2
I20260926 02:20:16.782979 18628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b: Bootstrap starting.
I20260926 02:20:16.786476 18628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:16.787828 18628 log.cc:824] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:16.789348 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:16.776640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.12.244.254:44931)
0926 02:20:16.777666 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:16.777677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:16.777721 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:20:16.779478 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:16.779505 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:16.780044 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:16.780502 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:16.781142 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.782450 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.784056 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.784926 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.785152 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:16.786450 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:16.786458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:16.786479 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:16.787244 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:16.787349 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:16.787371 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:16.787501 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:16.787757 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 02:20:16.788147 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":390,"threads_started":1}
I20260926 02:20:16.790988 18628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b: No bootstrap required, opened a new log
I20260926 02:20:16.793558 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:20:16.799347 18628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758eba72959e4f709e66da5705401f4b" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44931 } }
I20260926 02:20:16.799795 18628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:16.799916 18628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758eba72959e4f709e66da5705401f4b, State: Initialized, Role: FOLLOWER
I20260926 02:20:16.800617 18628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758eba72959e4f709e66da5705401f4b" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44931 } }
I20260926 02:20:16.800935 18628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:16.801110 18628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:16.801330 18628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:16.803751 18628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758eba72959e4f709e66da5705401f4b" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44931 } }
I20260926 02:20:16.804245 18628 leader_election.cc:302] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 758eba72959e4f709e66da5705401f4b; no voters:
I20260926 02:20:16.804833 18628 leader_election.cc:288] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:16.804948 18633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:16.806260 18633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [term 1 LEADER]: Becoming Leader. State: Replica: 758eba72959e4f709e66da5705401f4b, State: Running, Role: LEADER
I20260926 02:20:16.806783 18633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758eba72959e4f709e66da5705401f4b" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44931 } }
I20260926 02:20:16.807171 18628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:16.809737 18635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 758eba72959e4f709e66da5705401f4b. Latest consensus state: current_term: 1 leader_uuid: "758eba72959e4f709e66da5705401f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758eba72959e4f709e66da5705401f4b" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44931 } } }
I20260926 02:20:16.810214 18635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:16.812490 18634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "758eba72959e4f709e66da5705401f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758eba72959e4f709e66da5705401f4b" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44931 } } }
I20260926 02:20:16.812930 18634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:16.813441 18639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:16.818966 18639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:16.829646 18639 catalog_manager.cc:1347] Generated new cluster ID: 6f9e0945a7ab463f801d2e71be6a12a7
I20260926 02:20:16.829789 18639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:16.840448 18639 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:16.841843 18639 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:16.856566 18639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 758eba72959e4f709e66da5705401f4b: Generated new TSK 0
I20260926 02:20:16.857488 18639 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:20:16.897294 18658 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 02:20:16.897431 18658 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:20:16.898624 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:16.871473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.12.244.254:44931)
0926 02:20:16.871681 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:16.871688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:16.872072 (+ 384us) server_negotiation.cc:407] Connection header received
0926 02:20:16.872337 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:16.872341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:16.872424 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:16.872605 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:20:16.873120 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.873838 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.875262 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.875722 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.875863 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:16.876367 (+ 504us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:20:16.897818 (+ 21451us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:20:16.898255 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 02:20:16.898774 18657 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:16.871357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:44931 (local address 127.0.0.1:41738)
0926 02:20:16.871960 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:16.871976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:16.872153 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:16.872621 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:16.872626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:16.872638 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:20:16.872931 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:16.872940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.873992 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.874002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:16.875067 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:16.875078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:16.875908 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:16.875916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:16.876071 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:16.876106 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:20:16.898096 (+ 21990us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 02:20:16.898371 (+ 275us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.244.254:44931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":325,"threads_started":1}
W20260926 02:20:16.898770 18631 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 02:20:16.900224 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18562
I20260926 02:20:16.901453 18623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:16.995756 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18562
2026-09-26T02:20:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (759 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 02:20:17.052402 13267 mini_oidc.cc:152] Starting JWKS server
I20260926 02:20:17.054647 13267 webserver.cc:460] Webserver started at https://127.0.0.1:42151/ using document root <none> and password file <none>
I20260926 02:20:17.054904 13267 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:20:17.055688 13267 webserver.cc:460] Webserver started at http://127.0.0.1:41107/ using document root <none> and password file <none>
2026-09-26T02:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:17Z Disabled control of system clock
I20260926 02:20:17.085903 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46029
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:43911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:46029
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42151/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:20:17.428498 18665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:17.428824 18665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:17.428987 18665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:17.436460 18665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:17.436583 18665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:17.436671 18665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:17.436774 18665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:17.436866 18665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:20:17.436975 18665 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:20:17.437057 18665 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42151/jwks.json
I20260926 02:20:17.446645 18665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:46029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42151/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:17.448060 18665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:17.450001 18665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:17.462638 18670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:17.462805 18673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:17.462638 18671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:17.463516 18665 server_base.cc:1034] running on GCE node
I20260926 02:20:17.464355 18665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:17.465967 18665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:17.467221 18665 hybrid_clock.cc:648] HybridClock initialized: now 1790389217467153 us; error 63 us; skew 500 ppm
I20260926 02:20:17.467605 18665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:17.474634 18665 webserver.cc:460] Webserver started at http://127.12.244.254:40961/ using document root <none> and password file <none>
I20260926 02:20:17.475541 18665 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:17.475678 18665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:17.476075 18665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:17.479229 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "82d0b21a9c6546c0afe3d0e3868e5b60"
format_stamp: "Formatted at 2026-09-26 02:20:17 on dist-test-slave-qfmw"
server_key: "e9ffefd776ff98459bff3e286ed56bcb"
server_key_iv: "fa776cdac1657e1f890aeb3591a88265"
server_key_version: "encryptionkey@0"
I20260926 02:20:17.480125 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "82d0b21a9c6546c0afe3d0e3868e5b60"
format_stamp: "Formatted at 2026-09-26 02:20:17 on dist-test-slave-qfmw"
server_key: "e9ffefd776ff98459bff3e286ed56bcb"
server_key_iv: "fa776cdac1657e1f890aeb3591a88265"
server_key_version: "encryptionkey@0"
I20260926 02:20:17.485709 18665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:20:17.489643 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:17.491161 18665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 02:20:17.491371 18665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "82d0b21a9c6546c0afe3d0e3868e5b60"
format_stamp: "Formatted at 2026-09-26 02:20:17 on dist-test-slave-qfmw"
server_key: "e9ffefd776ff98459bff3e286ed56bcb"
server_key_iv: "fa776cdac1657e1f890aeb3591a88265"
server_key_version: "encryptionkey@0"
I20260926 02:20:17.491652 18665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:17.516487 18665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:17.517457 18665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:17.517740 18665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:17.537245 18665 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:46029
I20260926 02:20:17.537266 18730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:46029 every 8 connection(s)
I20260926 02:20:17.538697 18665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:17.542868 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:17.547581 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18665
I20260926 02:20:17.547801 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:17.548427 13267 external_mini_cluster.cc:1428] Setting key c3d5c5fd5cd5b26fb1d5140244ff41e1
I20260926 02:20:17.554411 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60: Bootstrap starting.
I20260926 02:20:17.557670 18731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:17.558789 18731 log.cc:824] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:17.562811 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60: No bootstrap required, opened a new log
I20260926 02:20:17.563210 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:17.550475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.12.244.254:46029)
0926 02:20:17.551558 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:17.551571 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:17.551619 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:20:17.553345 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:17.553371 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:17.553867 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:17.554284 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:17.554897 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:17.556379 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:17.557835 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:17.558555 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:17.558765 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:17.560164 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:17.560173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:17.560189 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:17.560939 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:17.561040 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:17.561065 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:17.561189 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:17.561421 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 02:20:17.561846 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":350,"threads_started":1}
I20260926 02:20:17.567512 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:20:17.571648 18731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46029 } }
I20260926 02:20:17.572084 18731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:17.572207 18731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d0b21a9c6546c0afe3d0e3868e5b60, State: Initialized, Role: FOLLOWER
I20260926 02:20:17.572861 18731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46029 } }
I20260926 02:20:17.573100 18731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:17.573257 18731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:17.573436 18731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:17.575822 18731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46029 } }
I20260926 02:20:17.576388 18731 leader_election.cc:302] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d0b21a9c6546c0afe3d0e3868e5b60; no voters:
I20260926 02:20:17.577036 18731 leader_election.cc:288] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:17.577190 18736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:17.577922 18736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [term 1 LEADER]: Becoming Leader. State: Replica: 82d0b21a9c6546c0afe3d0e3868e5b60, State: Running, Role: LEADER
I20260926 02:20:17.578630 18736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46029 } }
I20260926 02:20:17.579491 18731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:17.583461 18738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d0b21a9c6546c0afe3d0e3868e5b60. Latest consensus state: current_term: 1 leader_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46029 } } }
I20260926 02:20:17.584012 18738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:17.583803 18737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d0b21a9c6546c0afe3d0e3868e5b60" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46029 } } }
I20260926 02:20:17.584226 18737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:17.586335 18745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:17.591692 18745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:17.601691 18745 catalog_manager.cc:1347] Generated new cluster ID: 10c80d1546ae4ced813c7f69baebb7c3
I20260926 02:20:17.601837 18745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:17.619354 18745 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:17.621246 18745 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:17.637732 18745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d0b21a9c6546c0afe3d0e3868e5b60: Generated new TSK 0
I20260926 02:20:17.638973 18745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:20:17.669744 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:17.660951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.12.244.254:46029)
0926 02:20:17.661148 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:17.661154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:17.661523 (+ 369us) server_negotiation.cc:407] Connection header received
0926 02:20:17.661744 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:17.661751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:17.661846 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:17.662035 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:20:17.662524 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:17.663706 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:17.665169 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:17.665883 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:17.666014 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:17.666535 (+ 521us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:20:17.669065 (+ 2530us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42151/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:20:17.669416 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42151/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 02:20:17.669884 18734 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42151/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 02:20:17.671468 18760 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:17.660786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46029 (local address 127.0.0.1:54018)
0926 02:20:17.661402 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:17.661423 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:17.661551 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:17.662051 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:17.662054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:17.662064 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:20:17.662343 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:17.662351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:17.663880 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:17.663885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:17.665007 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:17.665020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:17.666083 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:17.666087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:17.666215 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:17.666251 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:20:17.669310 (+ 3059us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42151/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 02:20:17.671017 (+ 1707us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.244.254:46029: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42151/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":1350,"thread_start_us":340,"threads_started":1}
I20260926 02:20:17.672858 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18665
I20260926 02:20:17.674235 18726 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 02:20:17.681529 18761 webserver.cc:533] Webserver: sslize failed:
E20260926 02:20:17.681640 18761 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:20:17.766713 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18665
2026-09-26T02:20:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (770 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T02:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:17Z Disabled control of system clock
I20260926 02:20:17.851796 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46673
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:32779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:46673
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/testchainca.cert with env {}
W20260926 02:20:18.194073 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:18.194404 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:18.194492 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:18.201970 18766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:18.202085 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:18.202123 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:18.202171 18766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:18.202216 18766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:20:18.202272 18766 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:20:18.202318 18766 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 02:20:18.212997 18766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:32779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:46673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:18.214332 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:18.215670 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:18.228353 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:18.228422 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:18.228936 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:18.229156 18766 server_base.cc:1034] running on GCE node
I20260926 02:20:18.229988 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:18.231698 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:18.232954 18766 hybrid_clock.cc:648] HybridClock initialized: now 1790389218232879 us; error 55 us; skew 500 ppm
I20260926 02:20:18.233362 18766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:20:18.236231 18766 webserver.cc:460] Webserver started at http://127.12.244.254:45307/ using document root <none> and password file <none>
I20260926 02:20:18.237011 18766 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:18.237118 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:18.237407 18766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:18.240764 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "35a87838408141e8a93eceb2787f5373"
format_stamp: "Formatted at 2026-09-26 02:20:18 on dist-test-slave-qfmw"
server_key: "1f29acbc4c1c2c03d44a43a9d72ec07f"
server_key_iv: "01587f3de3d9da07d5cf3d64a37d4e7e"
server_key_version: "encryptionkey@0"
I20260926 02:20:18.241636 18766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "35a87838408141e8a93eceb2787f5373"
format_stamp: "Formatted at 2026-09-26 02:20:18 on dist-test-slave-qfmw"
server_key: "1f29acbc4c1c2c03d44a43a9d72ec07f"
server_key_iv: "01587f3de3d9da07d5cf3d64a37d4e7e"
server_key_version: "encryptionkey@0"
I20260926 02:20:18.247581 18766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 02:20:18.251343 18780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:18.253087 18766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:18.253327 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "35a87838408141e8a93eceb2787f5373"
format_stamp: "Formatted at 2026-09-26 02:20:18 on dist-test-slave-qfmw"
server_key: "1f29acbc4c1c2c03d44a43a9d72ec07f"
server_key_iv: "01587f3de3d9da07d5cf3d64a37d4e7e"
server_key_version: "encryptionkey@0"
I20260926 02:20:18.253582 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:18.289244 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:18.290181 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:18.290535 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:18.309506 18831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:46673 every 8 connection(s)
I20260926 02:20:18.309525 18766 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:46673
I20260926 02:20:18.311033 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:18.314215 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18766
I20260926 02:20:18.314433 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:18.315392 13267 external_mini_cluster.cc:1428] Setting key 3503869666360629fe606983fd04ea55
I20260926 02:20:18.315912 18832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:18.329766 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373: Bootstrap starting.
I20260926 02:20:18.330085 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:18.317582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.12.244.254:46673)
0926 02:20:18.318723 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:18.318736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:18.318779 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:20:18.321082 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:18.321109 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:18.321653 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:18.322109 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:18.322731 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:18.324147 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:18.325446 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:18.326128 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:18.326336 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:18.327660 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:18.327667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:18.327683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:20:18.328250 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:18.328344 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:20:18.328366 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:18.328493 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:18.328720 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 02:20:18.329054 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":380,"threads_started":1}
I20260926 02:20:18.334007 18832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:18.334502 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:20:18.335376 18832 log.cc:824] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:18.338737 18832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373: No bootstrap required, opened a new log
I20260926 02:20:18.347235 18832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a87838408141e8a93eceb2787f5373" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46673 } }
I20260926 02:20:18.347666 18832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:18.347766 18832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a87838408141e8a93eceb2787f5373, State: Initialized, Role: FOLLOWER
I20260926 02:20:18.348423 18832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a87838408141e8a93eceb2787f5373" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46673 } }
I20260926 02:20:18.348650 18832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:18.348773 18832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:18.348917 18832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:18.351709 18832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a87838408141e8a93eceb2787f5373" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46673 } }
I20260926 02:20:18.352241 18832 leader_election.cc:302] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a87838408141e8a93eceb2787f5373; no voters:
I20260926 02:20:18.352844 18832 leader_election.cc:288] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:18.353015 18837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:18.353801 18837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [term 1 LEADER]: Becoming Leader. State: Replica: 35a87838408141e8a93eceb2787f5373, State: Running, Role: LEADER
I20260926 02:20:18.354323 18837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a87838408141e8a93eceb2787f5373" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46673 } }
I20260926 02:20:18.355226 18832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:18.357460 18838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35a87838408141e8a93eceb2787f5373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a87838408141e8a93eceb2787f5373" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46673 } } }
I20260926 02:20:18.357439 18839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35a87838408141e8a93eceb2787f5373. Latest consensus state: current_term: 1 leader_uuid: "35a87838408141e8a93eceb2787f5373" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a87838408141e8a93eceb2787f5373" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46673 } } }
I20260926 02:20:18.358014 18838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:18.358035 18839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:18.359319 18846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:18.365263 18846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:18.375792 18846 catalog_manager.cc:1347] Generated new cluster ID: bbd04371603e4ea2b2aebe4223eb959a
I20260926 02:20:18.375941 18846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:18.391991 18846 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:18.393464 18846 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:18.403426 18846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35a87838408141e8a93eceb2787f5373: Generated new TSK 0
I20260926 02:20:18.404464 18846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:20:18.547312 18835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:18.424637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.12.244.254:46673)
0926 02:20:18.424850 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:18.424857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:18.425280 (+ 423us) server_negotiation.cc:407] Connection header received
0926 02:20:18.425504 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:18.425510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:18.425621 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:18.425799 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:20:18.426218 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:18.427433 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:18.428815 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:18.429534 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:18.429665 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:18.430208 (+ 543us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:20:18.546496 (+116288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:20:18.546958 (+ 462us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53212: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 02:20:18.547461 18835 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53212: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 02:20:18.547559 18861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:18.424482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46673 (local address 127.0.0.1:53212)
0926 02:20:18.425172 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:18.425189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:18.425319 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:18.425814 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:18.425817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:18.425826 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:20:18.426065 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:18.426072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:18.427576 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:18.427580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:18.428662 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:18.428674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:18.429745 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:18.429750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:18.429883 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:18.429919 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:20:18.546790 (+116871us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 02:20:18.547080 (+ 290us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.244.254:46673: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":385,"threads_started":1}
I20260926 02:20:18.548702 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18766
I20260926 02:20:18.549948 18827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:18.655933 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18766
2026-09-26T02:20:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (869 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (454 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (433 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:19 dist-test-slave-qfmw krb5kdc[18876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:19 dist-test-slave-qfmw krb5kdc[18876](info): set up 2 sockets
Sep 26 02:20:19 dist-test-slave-qfmw krb5kdc[18876](info): commencing operation
krb5kdc: starting...
W20260926 02:20:20.638414 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:20 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:20:20.860065 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34175
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:34175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:21.204872 18889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:21.205199 18889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:21.205305 18889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:21.212769 18889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:21.212901 18889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:21.212946 18889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:21.212998 18889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:21.213042 18889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:21.223793 18889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:34175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:21.225178 18889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:21.227273 18889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:21.240900 18897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:21.240949 18895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:21.240963 18894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:21.241736 18889 server_base.cc:1034] running on GCE node
I20260926 02:20:21.242545 18889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:21.244143 18889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:21.245352 18889 hybrid_clock.cc:648] HybridClock initialized: now 1790389221245252 us; error 89 us; skew 500 ppm
I20260926 02:20:21.245723 18889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:21 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:21.249697 18889 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:20:21.251497 18889 webserver.cc:460] Webserver started at http://127.12.244.254:45479/ using document root <none> and password file <none>
I20260926 02:20:21.252262 18889 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:21.252411 18889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:21.252776 18889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:21.256412 18889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10"
format_stamp: "Formatted at 2026-09-26 02:20:21 on dist-test-slave-qfmw"
server_key: "44d717f03917cdd022ae387d9975ce56"
server_key_iv: "27cbd4dd1946d84cc8d9c92f2554a5ab"
server_key_version: "encryptionkey@0"
I20260926 02:20:21.257333 18889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10"
format_stamp: "Formatted at 2026-09-26 02:20:21 on dist-test-slave-qfmw"
server_key: "44d717f03917cdd022ae387d9975ce56"
server_key_iv: "27cbd4dd1946d84cc8d9c92f2554a5ab"
server_key_version: "encryptionkey@0"
I20260926 02:20:21.263664 18889 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:20:21.267768 18904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:21.269418 18889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:21.269661 18889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10"
format_stamp: "Formatted at 2026-09-26 02:20:21 on dist-test-slave-qfmw"
server_key: "44d717f03917cdd022ae387d9975ce56"
server_key_iv: "27cbd4dd1946d84cc8d9c92f2554a5ab"
server_key_version: "encryptionkey@0"
I20260926 02:20:21.269909 18889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:21.307647 18889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:21.312947 18889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:21.313293 18889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:21.333564 18889 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:34175
I20260926 02:20:21.333598 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:34175 every 8 connection(s)
I20260926 02:20:21.335219 18889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:21.339288 18956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:21.343655 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18889
I20260926 02:20:21.343910 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:21.344671 13267 external_mini_cluster.cc:1428] Setting key 6efd3dda133de7fa08841257b35fe47c
I20260926 02:20:21.352020 18956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10: Bootstrap starting.
I20260926 02:20:21.356125 18956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:21.357834 18956 log.cc:824] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:21.361722 18956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10: No bootstrap required, opened a new log
Sep 26 02:20:21 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:21.373409 18956 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34175 } }
I20260926 02:20:21.373935 18956 raft_consensus.cc:374] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:21.373970 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:21.346835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.12.244.254:34175)
0926 02:20:21.347724 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:21.347737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:21.347780 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:20:21.350339 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:21.350369 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:21.350832 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:21.351307 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:21.353155 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:21.355010 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:21.357210 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:21.358117 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:21.358278 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:21.363189 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:21.363201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:21.363236 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:21.363301 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:21.368472 (+ 5171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:21.369449 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:21.369461 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:21.369476 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:21.369734 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:21.370158 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:21.370163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:21.370165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:21.371032 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:21.371403 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:21.371884 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 02:20:21.372428 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":447,"threads_started":1}
I20260926 02:20:21.374073 18956 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57b8cd71f1d466e8618a4b2c9d4ff10, State: Initialized, Role: FOLLOWER
I20260926 02:20:21.374687 18956 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34175 } }
I20260926 02:20:21.375015 18956 raft_consensus.cc:388] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:21.375211 18956 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:21.375411 18956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:21.377758 18956 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34175 } }
I20260926 02:20:21.378319 18956 leader_election.cc:302] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b57b8cd71f1d466e8618a4b2c9d4ff10; no voters:
I20260926 02:20:21.379148 18956 leader_election.cc:288] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:21.379285 18961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:21.380648 18961 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [term 1 LEADER]: Becoming Leader. State: Replica: b57b8cd71f1d466e8618a4b2c9d4ff10, State: Running, Role: LEADER
I20260926 02:20:21.381219 18961 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34175 } }
I20260926 02:20:21.381505 18956 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:21.384577 18962 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34175 } } }
I20260926 02:20:21.384547 18963 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57b8cd71f1d466e8618a4b2c9d4ff10. Latest consensus state: current_term: 1 leader_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b57b8cd71f1d466e8618a4b2c9d4ff10" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34175 } } }
I20260926 02:20:21.385077 18962 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:21.385098 18963 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:21.395536 18969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:21.402266 18969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:21.414106 18969 catalog_manager.cc:1347] Generated new cluster ID: 05903f817e92461eae031428879874f5
I20260926 02:20:21.414284 18969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:21.445642 18969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:21.447366 18969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:21.456166 18969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57b8cd71f1d466e8618a4b2c9d4ff10: Generated new TSK 0
I20260926 02:20:21.457217 18969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:20:21.522867 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34175
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:21.872256 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:21.872637 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:21.872802 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:21.880631 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:21.880831 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:21.881011 18982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:21.891521 18982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:21.892972 18982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:21.894686 18982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:21.909243 18987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:21.909152 18988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:21.909194 18990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:21.913118 18982 server_base.cc:1034] running on GCE node
I20260926 02:20:21.913875 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:21.915004 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:21.916218 18982 hybrid_clock.cc:648] HybridClock initialized: now 1790389221916147 us; error 67 us; skew 500 ppm
I20260926 02:20:21.916579 18982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:21 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:21.919974 18982 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:20:21.921667 18982 webserver.cc:460] Webserver started at http://127.12.244.193:40081/ using document root <none> and password file <none>
I20260926 02:20:21.922410 18982 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:21.922513 18982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:21.922780 18982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:21.925923 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "f5c7b51a454140979aeb8310d858ebd3"
format_stamp: "Formatted at 2026-09-26 02:20:21 on dist-test-slave-qfmw"
server_key: "bee308febdc4678eb9b8ba89437cddfa"
server_key_iv: "4c9c29d85baae9dc799c1e6d18e0fb86"
server_key_version: "encryptionkey@0"
I20260926 02:20:21.926787 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "f5c7b51a454140979aeb8310d858ebd3"
format_stamp: "Formatted at 2026-09-26 02:20:21 on dist-test-slave-qfmw"
server_key: "bee308febdc4678eb9b8ba89437cddfa"
server_key_iv: "4c9c29d85baae9dc799c1e6d18e0fb86"
server_key_version: "encryptionkey@0"
I20260926 02:20:21.932941 18982 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260926 02:20:21.937217 18997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:21.939021 18982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:20:21.939303 18982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "f5c7b51a454140979aeb8310d858ebd3"
format_stamp: "Formatted at 2026-09-26 02:20:21 on dist-test-slave-qfmw"
server_key: "bee308febdc4678eb9b8ba89437cddfa"
server_key_iv: "4c9c29d85baae9dc799c1e6d18e0fb86"
server_key_version: "encryptionkey@0"
I20260926 02:20:21.939558 18982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:21.957897 18982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:21.962561 18982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:21.962895 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:21.963982 18982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:21.965517 18982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:21.965622 18982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:21.965775 18982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:21.965853 18982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:22.000373 18982 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:35169
I20260926 02:20:22.000473 19109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:35169 every 8 connection(s)
I20260926 02:20:22.001755 18982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:22.006384 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 18982
I20260926 02:20:22.006592 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:22.007458 13267 external_mini_cluster.cc:1428] Setting key 94c922d497ee4da4939290a36956f7d0
Sep 26 02:20:22 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389221, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:22.026443 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:22.004385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:49045 (local address 127.12.244.254:34175)
0926 02:20:22.004859 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:22.004864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:22.005760 (+ 896us) server_negotiation.cc:407] Connection header received
0926 02:20:22.008524 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:22.008529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:22.008628 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:22.008751 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:22.010873 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.011658 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.014294 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.014703 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.014811 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:22.018905 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:22.018914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:22.018922 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:22.018970 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:22.022675 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:22.023591 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:22.023600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:22.023606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:22.023770 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:22.024273 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:22.024282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:22.024288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:22.024690 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:22.024848 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:22.025650 (+ 802us) server_negotiation.cc:299] Negotiation successful
0926 02:20:22.025926 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260926 02:20:22.027420 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:22.004838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34175 (local address 127.12.244.193:49045)
0926 02:20:22.005589 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:22.005628 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:22.008207 (+ 2579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:22.009029 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:22.009040 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:22.009776 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:22.010660 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:22.010691 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.011833 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.011840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:22.014105 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:22.014126 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.014957 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.014967 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:22.015364 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:22.016167 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:22.016207 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:22.018638 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:22.022859 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:22.022874 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:22.022892 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:22.023455 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:22.023917 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:22.023920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:22.023923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:22.024127 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:22.024826 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:22.024834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:22.025221 (+ 387us) client_negotiation.cc:769] Sending connection context
0926 02:20:22.025496 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 02:20:22.025921 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":290,"threads_started":1}
I20260926 02:20:22.029347 19110 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34175
I20260926 02:20:22.029711 19110 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:20:22.030540 19110 heartbeater.cc:511] Master 127.12.244.254:34175 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:20:22.033083 18921 ts_manager.cc:194] Registered new tserver with Master: f5c7b51a454140979aeb8310d858ebd3 (127.12.244.193:35169)
I20260926 02:20:22.034612 18921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:49045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:20:22.059430 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34175
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:22.398442 19116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:22.398823 19116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:22.398969 19116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:22.406592 19116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:22.406725 19116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:22.406847 19116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:22.417186 19116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:22.418499 19116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:22.419947 19116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:22.434216 19121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:22.434134 19124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:22.434176 19122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:22.435227 19116 server_base.cc:1034] running on GCE node
I20260926 02:20:22.435824 19116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:22.437003 19116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:22.438226 19116 hybrid_clock.cc:648] HybridClock initialized: now 1790389222438170 us; error 54 us; skew 500 ppm
I20260926 02:20:22.438581 19116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:22 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:22.441864 19116 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:20:22.443558 19116 webserver.cc:460] Webserver started at http://127.12.244.194:44193/ using document root <none> and password file <none>
I20260926 02:20:22.444321 19116 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:22.444471 19116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:22.444850 19116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:22.448084 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "1ff96e4047ae4c04973d553393ab74fc"
format_stamp: "Formatted at 2026-09-26 02:20:22 on dist-test-slave-qfmw"
server_key: "96b35e9b2db217384f9be98ff282c477"
server_key_iv: "7198588ec78d6f30b3f5b29c10e63a06"
server_key_version: "encryptionkey@0"
I20260926 02:20:22.448925 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "1ff96e4047ae4c04973d553393ab74fc"
format_stamp: "Formatted at 2026-09-26 02:20:22 on dist-test-slave-qfmw"
server_key: "96b35e9b2db217384f9be98ff282c477"
server_key_iv: "7198588ec78d6f30b3f5b29c10e63a06"
server_key_version: "encryptionkey@0"
I20260926 02:20:22.454556 19116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:20:22.458478 19131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:22.459995 19116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 02:20:22.460192 19116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "1ff96e4047ae4c04973d553393ab74fc"
format_stamp: "Formatted at 2026-09-26 02:20:22 on dist-test-slave-qfmw"
server_key: "96b35e9b2db217384f9be98ff282c477"
server_key_iv: "7198588ec78d6f30b3f5b29c10e63a06"
server_key_version: "encryptionkey@0"
I20260926 02:20:22.460412 19116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:22.502008 19116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:22.506618 19116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:22.506973 19116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:22.508086 19116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:22.509517 19116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:22.509624 19116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:22.509781 19116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:22.509853 19116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:22.544610 19116 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:37549
I20260926 02:20:22.544668 19243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:37549 every 8 connection(s)
I20260926 02:20:22.546005 19116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:22.551978 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 19116
I20260926 02:20:22.552148 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:22.552614 13267 external_mini_cluster.cc:1428] Setting key bc9974b107983d1265b1c3a5d8a8ee5d
Sep 26 02:20:22 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:22.567252 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:22.548694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:47935 (local address 127.12.244.254:34175)
0926 02:20:22.549142 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:22.549147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:22.550045 (+ 898us) server_negotiation.cc:407] Connection header received
0926 02:20:22.551957 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:22.551962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:22.552058 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:22.552172 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:22.553911 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.554764 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.556451 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.556853 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.556998 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:22.560747 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:22.560752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:22.560755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:22.560788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:22.563606 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:22.564684 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:22.564690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:22.564693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:22.564888 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:22.565374 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:22.565379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:22.565382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:22.565666 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:22.565833 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:22.566596 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 02:20:22.566838 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":246,"threads_started":1}
I20260926 02:20:22.568534 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:22.549173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34175 (local address 127.12.244.194:47935)
0926 02:20:22.549874 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:22.549920 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:22.551672 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:22.552442 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:22.552451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:22.553024 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:22.553737 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:22.553757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.554910 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.554921 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:22.556257 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:22.556269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:22.557094 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:22.557100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:22.557339 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:22.558078 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:22.558105 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:22.560496 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:22.563789 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:22.563803 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:22.563826 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:22.564517 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:22.565010 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:22.565016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:22.565020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:22.565229 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:22.565801 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:22.565812 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:22.566204 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 02:20:22.566503 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 02:20:22.567125 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":293,"threads_started":1}
I20260926 02:20:22.570292 19244 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34175
I20260926 02:20:22.570657 19244 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:22.571504 19244 heartbeater.cc:511] Master 127.12.244.254:34175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
I20260926 02:20:22.573117 18921 ts_manager.cc:194] Registered new tserver with Master: 1ff96e4047ae4c04973d553393ab74fc (127.12.244.194:37549)
I20260926 02:20:22.574054 18921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:47935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:20:22.601176 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34175
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:22.953596 19250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:22.953920 19250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:22.954092 19250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:22.961916 19250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:22.962100 19250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:22.962312 19250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:22.972612 19250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:22.974094 19250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:22.975562 19250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:22.989552 19258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:22.989507 19256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:22.989530 19255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:22.990692 19250 server_base.cc:1034] running on GCE node
I20260926 02:20:22.991506 19250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:22.992574 19250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:22.993818 19250 hybrid_clock.cc:648] HybridClock initialized: now 1790389222993760 us; error 43 us; skew 500 ppm
I20260926 02:20:22.994228 19250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:22 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:22.997550 19250 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:20:22.999203 19250 webserver.cc:460] Webserver started at http://127.12.244.195:41207/ using document root <none> and password file <none>
I20260926 02:20:23.000001 19250 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:23.000154 19250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:23.000492 19250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:23.003763 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "36be299e6fa848b993a8e48fccdc97c7"
format_stamp: "Formatted at 2026-09-26 02:20:22 on dist-test-slave-qfmw"
server_key: "319c87741e57236e7606a5910c4962ae"
server_key_iv: "bb47305c68154c65bf13b8c25caec950"
server_key_version: "encryptionkey@0"
I20260926 02:20:23.004714 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "36be299e6fa848b993a8e48fccdc97c7"
format_stamp: "Formatted at 2026-09-26 02:20:22 on dist-test-slave-qfmw"
server_key: "319c87741e57236e7606a5910c4962ae"
server_key_iv: "bb47305c68154c65bf13b8c25caec950"
server_key_version: "encryptionkey@0"
I20260926 02:20:23.010501 19250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:20:23.014658 19265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:23.016309 19250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:20:23.016530 19250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "36be299e6fa848b993a8e48fccdc97c7"
format_stamp: "Formatted at 2026-09-26 02:20:22 on dist-test-slave-qfmw"
server_key: "319c87741e57236e7606a5910c4962ae"
server_key_iv: "bb47305c68154c65bf13b8c25caec950"
server_key_version: "encryptionkey@0"
I20260926 02:20:23.016763 19250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:23.038743 19250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:23.038818 19110 heartbeater.cc:503] Master 127.12.244.254:34175 was elected leader, sending a full tablet report...
I20260926 02:20:23.044142 19250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:23.044497 19250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:23.045598 19250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:23.047037 19250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:23.047176 19250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 02:20:23.047362 19250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:23.047415 19250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:23.083307 19250 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:46567
I20260926 02:20:23.083369 19377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:46567 every 8 connection(s)
I20260926 02:20:23.084715 19250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:23.093232 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 19250
I20260926 02:20:23.093380 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:23.093852 13267 external_mini_cluster.cc:1428] Setting key 1bb6ad5e347d09445c2c8fbb26634884
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389222, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:23.107658 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.087654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:42329 (local address 127.12.244.254:34175)
0926 02:20:23.088119 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.088124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.089399 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 02:20:23.091602 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.091608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.091734 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.091864 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.093805 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.094816 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.096960 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.097414 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.097560 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.101552 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:23.101558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:23.101562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:23.101599 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:23.104451 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.105263 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.105268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.105270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.105441 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.105838 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.105842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.105844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.106124 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:23.106246 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:23.106968 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 02:20:23.107247 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":219,"threads_started":1}
I20260926 02:20:23.108842 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.088327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34175 (local address 127.12.244.195:42329)
0926 02:20:23.089148 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:23.089190 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:23.091276 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:23.092151 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:23.092162 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:23.092801 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:23.093593 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:23.093621 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.094994 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.094999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:23.096781 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:23.096795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.097667 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.097676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:23.097946 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.098730 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:23.098757 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:23.101282 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:23.104613 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:23.104622 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:23.104638 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:23.105128 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:23.105546 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:23.105549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:23.105551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:23.105736 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:23.106248 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:23.106255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:23.106635 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 02:20:23.106884 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 02:20:23.107399 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":314,"threads_started":1}
I20260926 02:20:23.110625 19378 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34175
I20260926 02:20:23.110984 19378 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:23.111779 19378 heartbeater.cc:511] Master 127.12.244.254:34175 requested a full tablet report, sending...
I20260926 02:20:23.113479 18921 ts_manager.cc:194] Registered new tserver with Master: 36be299e6fa848b993a8e48fccdc97c7 (127.12.244.195:46567)
I20260926 02:20:23.114442 18921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:42329
I20260926 02:20:23.122664 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/test-admin.keytab.
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:23.150090 13267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 02:20:23.155926 19386 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:23.153180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.12.244.193:35169)
0926 02:20:23.153898 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.153906 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.153935 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:20:23.153996 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.154003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.154234 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.154371 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.155480 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32806: BlockingRecv error: recv got EOF from 127.0.0.1:32806 (error 108)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":362,"threads_started":1}
W20260926 02:20:23.156057 13267 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:23.158227 13267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:23.159708 13267 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:20:23.178251 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.161495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.12.244.193:35169)
0926 02:20:23.161723 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.161730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.161901 (+ 171us) server_negotiation.cc:407] Connection header received
0926 02:20:23.162102 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.162106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.162169 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.162256 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.163349 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.164499 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.165910 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.166514 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.166631 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.171085 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:23.171167 (+ 82us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:23.171188 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:23.171236 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:23.174720 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.175893 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.175902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.175905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.176055 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.176489 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.176494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.176496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.176920 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:23.177168 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:23.177578 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 02:20:23.177981 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 02:20:23.184978 19386 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:23.183039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.12.244.193:35169)
0926 02:20:23.183295 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.183301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.183537 (+ 236us) server_negotiation.cc:407] Connection header received
0926 02:20:23.183710 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.183715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.183799 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.183885 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.184723 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32834: BlockingRecv error: recv got EOF from 127.0.0.1:32834 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260926 02:20:23.185181 13267 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:23.186825 13267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:23.188585 13267 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:20:23.203832 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.190371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.12.244.193:35169)
0926 02:20:23.190596 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.190602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.190753 (+ 151us) server_negotiation.cc:407] Connection header received
0926 02:20:23.191008 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.191013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.191075 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.191248 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.192255 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.193006 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.194251 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.194674 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.194807 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.198645 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:23.198650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:23.198654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:23.198706 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:23.201300 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.202058 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.202062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.202064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.202189 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.202613 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.202616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.202618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.202923 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:23.203181 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:23.203443 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 02:20:23.203608 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 02:20:23.208676 19386 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:23.206820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.12.244.193:35169)
0926 02:20:23.207042 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.207048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.207293 (+ 245us) server_negotiation.cc:407] Connection header received
0926 02:20:23.207493 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.207498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.207566 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.207669 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.208373 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32842: BlockingRecv error: recv got EOF from 127.0.0.1:32842 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 02:20:23.208849 13267 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790389169147784-13267-0/krb5kdc/krb5cc' of type FILE
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:23.210530 13267 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:23.211947 13267 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:20:23.226887 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.213331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.12.244.193:35169)
0926 02:20:23.213595 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.213602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.213886 (+ 284us) server_negotiation.cc:407] Connection header received
0926 02:20:23.214068 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.214072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.214129 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.214238 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.215273 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.215997 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.217255 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.217665 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.217801 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.221545 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:23.221550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:23.221553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:23.221589 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:23.224408 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.225105 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.225109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.225111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.225280 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.225605 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.225608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.225610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.225891 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:23.226107 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:23.226323 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 02:20:23.226511 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 02:20:23.228612 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18982
I20260926 02:20:23.241056 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.229352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.12.244.193:35169)
0926 02:20:23.229530 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.229534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.229897 (+ 363us) server_negotiation.cc:407] Connection header received
0926 02:20:23.229974 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.229978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.230033 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.230111 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.231079 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.231851 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.233062 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.233539 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.233640 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.235734 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:23.235739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:23.235742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:23.235773 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:23.238572 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.239464 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.239469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.239473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.239611 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.239939 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.239942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.239944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.240189 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:23.240400 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:23.240623 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 02:20:23.240789 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:20:23.241657 19105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:23.326745 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18982
I20260926 02:20:23.343235 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 19116
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:20:23.360108 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.344308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.12.244.194:37549)
0926 02:20:23.344942 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.344957 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.344985 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:20:23.345047 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.345054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.345280 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.345428 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.346426 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.347576 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.348868 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.349438 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.349543 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.353624 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:23.353639 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:23.353656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:23.353703 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:23.356849 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.357724 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.357732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.357734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.357931 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.358426 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.358430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.358432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.358858 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:23.359136 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:23.359541 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 02:20:23.359753 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":313,"threads_started":1}
I20260926 02:20:23.361529 19239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:23.449236 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 19116
I20260926 02:20:23.465349 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 19250
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[18876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:20:23.482486 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:23.466392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.12.244.195:46567)
0926 02:20:23.467038 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:23.467045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:23.467076 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:20:23.467212 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:23.467219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:23.467447 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:23.467579 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:23.468655 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.469701 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.470945 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:23.471652 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:23.471757 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:23.475990 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:23.476006 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:23.476024 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:23.476072 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:23.479202 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.480066 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.480075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.480077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.480238 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:23.480809 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:23.480814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:23.480816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:23.481309 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:23.481570 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:23.481856 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 02:20:23.482087 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":300,"threads_started":1}
I20260926 02:20:23.484059 19373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:23.567430 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 19250
I20260926 02:20:23.583714 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 18889
I20260926 02:20:23.584957 18951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:23.690305 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 18889
2026-09-26T02:20:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4152 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[19406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[19406](info): set up 2 sockets
Sep 26 02:20:23 dist-test-slave-qfmw krb5kdc[19406](info): commencing operation
krb5kdc: starting...
W20260926 02:20:24.806500 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:24 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab.
Entry for principal oryx/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab.
I20260926 02:20:25.044021 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33863
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab
--principal=oryx/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:33863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:25.390748 19419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:25.391178 19419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:25.391309 19419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:25.398937 19419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:25.399075 19419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:25.399216 19419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:25.399318 19419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:25.399415 19419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:25.410245 19419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:33863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab
--principal=oryx/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:25.412005 19419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:25.413785 19419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:25.426762 19424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:25.427052 19427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:25.426795 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:25.427739 19419 server_base.cc:1034] running on GCE node
I20260926 02:20:25.428608 19419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:25.430190 19419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:25.431459 19419 hybrid_clock.cc:648] HybridClock initialized: now 1790389225431404 us; error 48 us; skew 500 ppm
I20260926 02:20:25.431881 19419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:25 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389225, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:25.435384 19419 init.cc:377] Logged in from keytab as oryx/127.12.244.254@KRBTEST.COM (short username oryx)
I20260926 02:20:25.437028 19419 webserver.cc:460] Webserver started at http://127.12.244.254:37023/ using document root <none> and password file <none>
I20260926 02:20:25.437789 19419 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:25.437958 19419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:25.438290 19419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:25.441582 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "4926ed7f7fde4132b6b7a411a01bb9df"
format_stamp: "Formatted at 2026-09-26 02:20:25 on dist-test-slave-qfmw"
server_key: "532591282f09e0c0df0edceb260b7bf5"
server_key_iv: "c01ea2b83c7ac45b50b1aec006e1ee31"
server_key_version: "encryptionkey@0"
I20260926 02:20:25.442473 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "4926ed7f7fde4132b6b7a411a01bb9df"
format_stamp: "Formatted at 2026-09-26 02:20:25 on dist-test-slave-qfmw"
server_key: "532591282f09e0c0df0edceb260b7bf5"
server_key_iv: "c01ea2b83c7ac45b50b1aec006e1ee31"
server_key_version: "encryptionkey@0"
I20260926 02:20:25.448629 19419 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 02:20:25.453009 19434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:25.454911 19419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:20:25.455245 19419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "4926ed7f7fde4132b6b7a411a01bb9df"
format_stamp: "Formatted at 2026-09-26 02:20:25 on dist-test-slave-qfmw"
server_key: "532591282f09e0c0df0edceb260b7bf5"
server_key_iv: "c01ea2b83c7ac45b50b1aec006e1ee31"
server_key_version: "encryptionkey@0"
I20260926 02:20:25.455541 19419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:25.472991 19419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:25.479557 19419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:25.479902 19419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:25.500855 19419 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:33863
I20260926 02:20:25.500880 19486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:33863 every 8 connection(s)
I20260926 02:20:25.502385 19419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:25.506382 19487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:25.507136 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 19419
I20260926 02:20:25.507344 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:25.508149 13267 external_mini_cluster.cc:1428] Setting key 790fbb020523caeaf524f6c10c2151df
I20260926 02:20:25.521767 19487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: Bootstrap starting.
Sep 26 02:20:25 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:25.525553 19487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:25.526669 19487 log.cc:824] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:25.530216 19487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: No bootstrap required, opened a new log
I20260926 02:20:25.533787 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:25.510480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.12.244.254:33863)
0926 02:20:25.511408 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:25.511421 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:25.511467 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:20:25.513323 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:25.513355 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:25.513831 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:25.514226 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:25.516187 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:25.517589 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:25.519501 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:25.520240 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:25.520390 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:25.525046 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:25.525054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:25.525072 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:25.525118 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:25.528708 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:25.529530 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:25.529540 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:25.529552 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:25.529771 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:25.530192 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:25.530203 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:25.530208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:25.530973 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:25.531311 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:25.531723 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 02:20:25.532198 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":416,"threads_started":1}
I20260926 02:20:25.539719 19487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } }
I20260926 02:20:25.540103 19487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:25.540181 19487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4926ed7f7fde4132b6b7a411a01bb9df, State: Initialized, Role: FOLLOWER
I20260926 02:20:25.540802 19487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } }
I20260926 02:20:25.541095 19487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:25.541234 19487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:25.541430 19487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:25.543746 19487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } }
I20260926 02:20:25.544344 19487 leader_election.cc:302] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4926ed7f7fde4132b6b7a411a01bb9df; no voters:
I20260926 02:20:25.545145 19487 leader_election.cc:288] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:25.545413 19492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:25.546864 19492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 1 LEADER]: Becoming Leader. State: Replica: 4926ed7f7fde4132b6b7a411a01bb9df, State: Running, Role: LEADER
I20260926 02:20:25.547484 19492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } }
I20260926 02:20:25.547647 19487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:25.550949 19493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } }
I20260926 02:20:25.551014 19494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4926ed7f7fde4132b6b7a411a01bb9df. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } }
I20260926 02:20:25.551425 19493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:25.551481 19494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:25.561968 19501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:25.566916 19501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:25.577036 19501 catalog_manager.cc:1347] Generated new cluster ID: f7663f0c98dd4b1c96e53242122e188c
I20260926 02:20:25.577164 19501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:25.593976 19501 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:25.596266 19501 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:25.614115 19501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: Generated new TSK 0
I20260926 02:20:25.615533 19501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab.
Entry for principal oryx/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab.
I20260926 02:20:25.678825 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33863
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab
--principal=oryx/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:26.026393 19513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:26.026705 19513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:26.026865 19513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:26.034581 19513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:26.034708 19513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:26.034868 19513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:26.045233 19513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab
--principal=oryx/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:26.046602 19513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:26.048110 19513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:26.062704 19521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:26.062680 19519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:26.062922 19518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:26.063642 19513 server_base.cc:1034] running on GCE node
I20260926 02:20:26.064260 19513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:26.065253 19513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:26.066502 19513 hybrid_clock.cc:648] HybridClock initialized: now 1790389226066457 us; error 50 us; skew 500 ppm
I20260926 02:20:26.066854 19513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:26 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389226, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:26.070299 19513 init.cc:377] Logged in from keytab as oryx/127.12.244.193@KRBTEST.COM (short username oryx)
I20260926 02:20:26.071982 19513 webserver.cc:460] Webserver started at http://127.12.244.193:41767/ using document root <none> and password file <none>
I20260926 02:20:26.072736 19513 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:26.072890 19513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:26.073211 19513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:26.076346 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "a5ee8b1e6a49c09f90ef66714e663026"
server_key_iv: "312b145cdca954b50ae4355336e8f898"
server_key_version: "encryptionkey@0"
I20260926 02:20:26.077208 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "a5ee8b1e6a49c09f90ef66714e663026"
server_key_iv: "312b145cdca954b50ae4355336e8f898"
server_key_version: "encryptionkey@0"
I20260926 02:20:26.082984 19513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:20:26.086948 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:26.088831 19513 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 02:20:26.089079 19513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "a5ee8b1e6a49c09f90ef66714e663026"
server_key_iv: "312b145cdca954b50ae4355336e8f898"
server_key_version: "encryptionkey@0"
I20260926 02:20:26.089388 19513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:26.125017 19513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:26.129473 19513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:26.129725 19513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:26.130740 19513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:26.132208 19513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:26.132282 19513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:26.132377 19513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:26.132426 19513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:26.167305 19513 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45503
I20260926 02:20:26.167328 19642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45503 every 8 connection(s)
I20260926 02:20:26.168718 19513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:26.171307 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 19513
I20260926 02:20:26.171546 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:26.172340 13267 external_mini_cluster.cc:1428] Setting key 8fc4a1344063eab5bac54c5b644c1a0c
Sep 26 02:20:26 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389226, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:26.191771 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:26.171776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:54709 (local address 127.12.244.254:33863)
0926 02:20:26.172236 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:26.172241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:26.173558 (+ 1317us) server_negotiation.cc:407] Connection header received
0926 02:20:26.175551 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:26.175556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:26.175659 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:26.175781 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:26.177407 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.178138 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.180137 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.180556 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.180688 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:26.185237 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:26.185245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:26.185249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:26.185286 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:26.188030 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:26.188982 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:26.188991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:26.188997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:26.189168 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:26.189643 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:26.189653 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:26.189659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:26.190106 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:26.190303 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:26.191044 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 02:20:26.191328 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":257,"threads_started":1}
I20260926 02:20:26.192579 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:26.172490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:54709)
0926 02:20:26.173398 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:26.173442 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:26.175264 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:26.176056 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:26.176066 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:26.176611 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:26.177220 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:26.177237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.178318 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.178322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:26.179923 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:26.179941 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.180832 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.180841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:26.181193 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:26.182016 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:26.182058 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:26.184952 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:26.188220 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:26.188239 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:26.188267 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:26.188785 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:26.189300 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:26.189304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:26.189306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:26.189497 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:26.190259 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:26.190267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:26.190635 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 02:20:26.190873 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 02:20:26.191379 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for oryx/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:20:26.194454 19643 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33863
I20260926 02:20:26.194811 19643 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:26.195712 19643 heartbeater.cc:511] Master 127.12.244.254:33863 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.194@KRBTEST.COM" created.
I20260926 02:20:26.198218 19451 ts_manager.cc:194] Registered new tserver with Master: f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193:45503)
I20260926 02:20:26.199913 19451 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:54709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab.
Entry for principal oryx/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab.
I20260926 02:20:26.220397 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33863
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab
--principal=oryx/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:26.565284 19649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:26.565600 19649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:26.565757 19649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:26.573354 19649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:26.573480 19649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:26.573663 19649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:26.583732 19649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab
--principal=oryx/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:26.585148 19649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:26.586652 19649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:26.600831 19654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:26.600867 19657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:26.600911 19655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:26.601732 19649 server_base.cc:1034] running on GCE node
I20260926 02:20:26.602388 19649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:26.603499 19649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:26.604723 19649 hybrid_clock.cc:648] HybridClock initialized: now 1790389226604650 us; error 64 us; skew 500 ppm
I20260926 02:20:26.605119 19649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:26 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389226, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:26.608745 19649 init.cc:377] Logged in from keytab as oryx/127.12.244.194@KRBTEST.COM (short username oryx)
I20260926 02:20:26.610491 19649 webserver.cc:460] Webserver started at http://127.12.244.194:46043/ using document root <none> and password file <none>
I20260926 02:20:26.611472 19649 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:26.611647 19649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:26.611989 19649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:26.616170 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "f6a302db476745678497bc364527197a"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "8d71e0ec492c71734eacbb6e1fefcab6"
server_key_iv: "35c357efc7cf504c9b647cced3f7a92f"
server_key_version: "encryptionkey@0"
I20260926 02:20:26.617511 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "f6a302db476745678497bc364527197a"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "8d71e0ec492c71734eacbb6e1fefcab6"
server_key_iv: "35c357efc7cf504c9b647cced3f7a92f"
server_key_version: "encryptionkey@0"
I20260926 02:20:26.624100 19649 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:20:26.628434 19664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:26.629977 19649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260926 02:20:26.630199 19649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "f6a302db476745678497bc364527197a"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "8d71e0ec492c71734eacbb6e1fefcab6"
server_key_iv: "35c357efc7cf504c9b647cced3f7a92f"
server_key_version: "encryptionkey@0"
I20260926 02:20:26.630434 19649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:26.645582 19649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:26.650269 19649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:26.650585 19649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:26.651726 19649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:26.653077 19649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:26.653183 19649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:26.653332 19649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:26.653414 19649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:26.688087 19649 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:39225
I20260926 02:20:26.688134 19777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:39225 every 8 connection(s)
I20260926 02:20:26.689424 19649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:26.692171 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 19649
I20260926 02:20:26.692358 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:26.692945 13267 external_mini_cluster.cc:1428] Setting key a75bcac663065b596486914435c5e09c
Sep 26 02:20:26 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389226, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:26.717345 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:26.692198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:52837 (local address 127.12.244.254:33863)
0926 02:20:26.692889 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:26.692896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:26.696727 (+ 3831us) server_negotiation.cc:407] Connection header received
0926 02:20:26.696800 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:26.696808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:26.696908 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:26.697017 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:26.699603 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.700810 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.703754 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.704504 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.704665 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:26.709719 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:26.709724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:26.709728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:26.709768 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:26.712815 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:26.714158 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:26.714163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:26.714166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:26.714352 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:26.715037 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:26.715042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:26.715044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:26.715405 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:26.715540 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:26.716757 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0926 02:20:26.717047 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":412,"threads_started":1}
WARNING: no policy specified for oryx/127.12.244.195@KRBTEST.COM; defaulting to no policy
I20260926 02:20:26.718878 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:26.692898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:52837)
0926 02:20:26.693960 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:26.694010 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:26.696337 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:26.697524 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:26.697540 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:26.698401 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:26.699365 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:26.699396 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.701035 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.701043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:26.703506 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:26.703528 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:26.704821 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:26.704832 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:26.705202 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:26.706044 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:26.706081 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:26.709340 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:26.713046 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:26.713060 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:26.713086 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:26.713939 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:26.714510 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:26.714520 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:26.714528 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:26.714856 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:26.715554 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:26.715565 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:26.716128 (+ 563us) client_negotiation.cc:769] Sending connection context
0926 02:20:26.716569 (+ 441us) client_negotiation.cc:240] Negotiation successful
0926 02:20:26.717253 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":380,"threads_started":1}
I20260926 02:20:26.720935 19778 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33863
I20260926 02:20:26.721350 19778 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.195@KRBTEST.COM" created.
I20260926 02:20:26.722224 19778 heartbeater.cc:511] Master 127.12.244.254:33863 requested a full tablet report, sending...
I20260926 02:20:26.724102 19451 ts_manager.cc:194] Registered new tserver with Master: f6a302db476745678497bc364527197a (127.12.244.194:39225)
I20260926 02:20:26.725401 19451 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:52837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab.
Entry for principal oryx/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab.
I20260926 02:20:26.752664 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33863
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab
--principal=oryx/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:27.111205 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:27.111584 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:27.111792 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:27.119544 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:27.119748 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:27.119882 19784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:27.130398 19784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab
--principal=oryx/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:27.131871 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:27.133236 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:27.148415 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:27.148442 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:27.148525 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:27.149333 19784 server_base.cc:1034] running on GCE node
I20260926 02:20:27.149999 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:27.151073 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:27.152428 19784 hybrid_clock.cc:648] HybridClock initialized: now 1790389227152369 us; error 32 us; skew 500 ppm
I20260926 02:20:27.152848 19784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:27 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389227, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:27.156656 19784 init.cc:377] Logged in from keytab as oryx/127.12.244.195@KRBTEST.COM (short username oryx)
I20260926 02:20:27.158423 19784 webserver.cc:460] Webserver started at http://127.12.244.195:36215/ using document root <none> and password file <none>
I20260926 02:20:27.159368 19784 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:27.159531 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:27.159932 19784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:27.163291 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "67b6992471b24ce0b726b833157df909"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "2c24ce633c5725f000650f32eb7bb266"
server_key_iv: "1e6375b3d8116cc47a29037ee80d5bff"
server_key_version: "encryptionkey@0"
I20260926 02:20:27.164227 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "67b6992471b24ce0b726b833157df909"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "2c24ce633c5725f000650f32eb7bb266"
server_key_iv: "1e6375b3d8116cc47a29037ee80d5bff"
server_key_version: "encryptionkey@0"
I20260926 02:20:27.171324 19784 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 02:20:27.176379 19799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:27.178256 19784 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:20:27.178452 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "67b6992471b24ce0b726b833157df909"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "2c24ce633c5725f000650f32eb7bb266"
server_key_iv: "1e6375b3d8116cc47a29037ee80d5bff"
server_key_version: "encryptionkey@0"
I20260926 02:20:27.178660 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:27.200676 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:27.203234 19643 heartbeater.cc:503] Master 127.12.244.254:33863 was elected leader, sending a full tablet report...
I20260926 02:20:27.206022 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:27.206348 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:27.207518 19784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:27.209070 19784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:27.209180 19784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:27.209336 19784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:27.209411 19784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:27.245415 19784 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:36613
I20260926 02:20:27.245474 19912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:36613 every 8 connection(s)
I20260926 02:20:27.246797 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:27.256819 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 19784
I20260926 02:20:27.257015 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:27.257601 13267 external_mini_cluster.cc:1428] Setting key 060ee449167d0fda2a4f2518c151984c
Sep 26 02:20:27 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389227, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:27.270084 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.249828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:59121 (local address 127.12.244.254:33863)
0926 02:20:27.250347 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:27.250354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:27.251484 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 02:20:27.253388 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:27.253393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:27.253486 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:27.253597 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:27.255411 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.256212 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.258631 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.259291 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.259444 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.263454 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:27.263459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:27.263463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:27.263501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:27.266406 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.267375 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.267380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.267382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.267514 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.268039 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.268044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.268046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.268345 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:27.268483 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:27.269541 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0926 02:20:27.269807 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":244,"threads_started":1}
I20260926 02:20:27.271277 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.250477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:59121)
0926 02:20:27.251306 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:27.251357 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:27.253070 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:27.253834 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:27.253890 (+ 56us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:27.254467 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:27.255182 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.255212 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.256405 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.256410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.258459 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.258474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.259574 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.259582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.259834 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.260586 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:27.260617 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:27.263077 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:27.266592 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.266601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.266618 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.267218 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.267635 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.267639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.267642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.267865 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.268505 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:27.268514 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:27.269095 (+ 581us) client_negotiation.cc:769] Sending connection context
0926 02:20:27.269416 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 02:20:27.269890 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":298,"threads_started":1}
I20260926 02:20:27.272926 19913 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33863
I20260926 02:20:27.273324 19913 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:27.274214 19913 heartbeater.cc:511] Master 127.12.244.254:33863 requested a full tablet report, sending...
I20260926 02:20:27.275992 19450 ts_manager.cc:194] Registered new tserver with Master: 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:27.277007 19450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:59121
I20260926 02:20:27.286583 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.244.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.253.keytab.
Entry for principal oryx/127.12.244.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.253.keytab.
I20260926 02:20:27.338557 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:35731
--webserver_interface=127.12.244.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.253.keytab
--principal=oryx/127.12.244.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:33863,127.12.244.253:35731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:27.691174 19919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:27.691545 19919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:27.691699 19919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:27.699496 19919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:27.699663 19919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:27.699759 19919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:27.699841 19919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:27.699923 19919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:27.710173 19919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:33863,127.12.244.253:35731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.253.keytab
--principal=oryx/127.12.244.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:35731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.244.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
W20260926 02:20:27.710631 19919 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.244.254:33863,127.12.244.253:35731'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 02:20:27.711807 19919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:27.713258 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:27.726059 19927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:27.726096 19925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:27.726022 19924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:27.727131 19919 server_base.cc:1034] running on GCE node
I20260926 02:20:27.727962 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:27.728749 19778 heartbeater.cc:503] Master 127.12.244.254:33863 was elected leader, sending a full tablet report...
I20260926 02:20:27.729740 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:27.731011 19919 hybrid_clock.cc:648] HybridClock initialized: now 1790389227730926 us; error 61 us; skew 500 ppm
I20260926 02:20:27.731401 19919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:27 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389227, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:27.740630 19919 init.cc:377] Logged in from keytab as oryx/127.12.244.253@KRBTEST.COM (short username oryx)
I20260926 02:20:27.742415 19919 webserver.cc:460] Webserver started at http://127.12.244.253:38821/ using document root <none> and password file <none>
I20260926 02:20:27.743314 19919 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:27.743460 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:27.743793 19919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:27.746879 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/instance:
uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "122647edc149aeb2b9cb062792b4ce8e"
server_key_iv: "95cded08a38005e656506785dd7830f0"
server_key_version: "encryptionkey@0"
I20260926 02:20:27.747798 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal/instance:
uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "122647edc149aeb2b9cb062792b4ce8e"
server_key_iv: "95cded08a38005e656506785dd7830f0"
server_key_version: "encryptionkey@0"
I20260926 02:20:27.753592 19919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 02:20:27.757519 19934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:27.759140 19919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 02:20:27.759343 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "122647edc149aeb2b9cb062792b4ce8e"
server_key_iv: "95cded08a38005e656506785dd7830f0"
server_key_version: "encryptionkey@0"
I20260926 02:20:27.759629 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:27.774406 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:27.779034 19919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:27.779421 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:27.799036 19919 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.253:35731
I20260926 02:20:27.799062 19985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.253:35731 every 8 connection(s)
I20260926 02:20:27.800493 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
I20260926 02:20:27.801075 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 19919
I20260926 02:20:27.805548 19986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:27.814738 19986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:20:27 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.253@KRBTEST.COM
Sep 26 02:20:27 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389227, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.253@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:27.827665 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.803606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.12.244.253:35731)
0926 02:20:27.804430 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:27.804454 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:27.804507 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:20:27.806790 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:27.806823 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:27.807404 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:27.807902 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:27.809342 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.811468 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.812846 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.813936 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.814127 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.818889 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:27.818902 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:27.818923 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:27.818980 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:27.822928 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.823707 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.823721 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.823728 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.823908 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.824347 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.824352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.824354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.824901 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:27.825265 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:27.825751 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 02:20:27.826197 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":341,"threads_started":1}
I20260926 02:20:27.833514 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.817504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.12.244.254:33863)
0926 02:20:27.818089 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:27.818097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:27.818729 (+ 632us) server_negotiation.cc:407] Connection header received
0926 02:20:27.818867 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:27.818872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:27.818961 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:27.819079 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:27.820612 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.821862 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.823456 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.823865 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.824022 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.827855 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:27.827862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:27.827867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:27.827914 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:27.830618 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.831454 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.831458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.831460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.831591 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.831987 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.831991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.831993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.832353 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:27.832516 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:27.833074 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 02:20:27.833283 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":336,"threads_started":1}
I20260926 02:20:27.833549 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.817503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:38998)
0926 02:20:27.818512 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:27.818563 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:27.818726 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:27.819185 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:27.819190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:27.820059 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:27.820380 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.820389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.822011 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.822016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.823284 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.823300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.824068 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.824075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.824257 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.824931 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:27.824949 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:27.827635 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:27.830782 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.830794 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.830801 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.831325 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.831688 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.831691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.831694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.831833 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.832533 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:27.832539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:27.832786 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 02:20:27.832954 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 02:20:27.833233 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":599,"threads_started":1}
I20260926 02:20:27.835409 19986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:20:27 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389227, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.253@KRBTEST.COM for oryx/127.12.244.253@KRBTEST.COM
I20260926 02:20:27.848397 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.836210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.0.0.1:50560)
0926 02:20:27.836409 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:27.836424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:27.836523 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:27.836804 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:27.836807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:27.837179 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:27.837317 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.837325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.838315 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.838319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.839286 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.839297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.840005 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.840010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.840138 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.840711 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:27.840728 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:27.842846 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:27.845922 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.845926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.845928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.846319 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.846670 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.846673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.846676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.846799 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.847518 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:27.847522 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:27.847730 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 02:20:27.847857 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 02:20:27.848076 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:20:27.848644 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.836201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.12.244.253:35731)
0926 02:20:27.836386 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:27.836391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:27.836503 (+ 112us) server_negotiation.cc:407] Connection header received
0926 02:20:27.836607 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:27.836611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:27.836664 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:27.836771 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:27.837439 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.838171 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.839445 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.839855 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.839961 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.843025 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:27.843030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:27.843033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:27.843064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:27.845755 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.846441 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.846445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.846448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.846573 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.846956 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.846960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.846962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.847330 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:27.847463 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:27.847953 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 02:20:27.848283 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":58}
I20260926 02:20:27.853585 19986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Bootstrap starting.
I20260926 02:20:27.856992 19986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:27.858145 19986 log.cc:824] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:27.861301 19986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: No bootstrap required, opened a new log
I20260926 02:20:27.868759 19986 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } }
I20260926 02:20:27.869110 19986 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:27.869190 19986 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b2df03947a42809d10cbfcbbeb1bdb, State: Initialized, Role: FOLLOWER
I20260926 02:20:27.869787 19986 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } }
I20260926 02:20:27.870654 19993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } } }
I20260926 02:20:27.870998 19993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:20:27.871851 19986 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:20:27.881220 20004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:20:27.881418 20004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:20:27.884301 19919 master_runner.cc:428] Detected that this master c8b2df03947a42809d10cbfcbbeb1bdb is joining an existing cluster
I20260926 02:20:27.884421 19919 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.244.253:35731
I20260926 02:20:27.905544 19993 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:27.905918 19993 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } }
I20260926 02:20:27.907578 19993 leader_election.cc:288] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863)
I20260926 02:20:27.908586 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.890537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.12.244.253:35731)
0926 02:20:27.890774 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:27.890782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:27.891540 (+ 758us) server_negotiation.cc:407] Connection header received
0926 02:20:27.891806 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:27.891814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:27.891914 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:27.891995 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:27.893862 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.894808 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.896796 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.897510 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.897609 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.900339 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:27.900347 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:27.900352 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:27.900393 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:27.904456 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.905353 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.905359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.905362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.905514 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.905958 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.905965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.905971 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.906382 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:27.906569 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:27.907220 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 02:20:27.907518 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:20:27.908556 19461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" is_pre_election: true
I20260926 02:20:27.908872 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.890644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.0.0.1:50576)
0926 02:20:27.891440 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:27.891467 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:27.891645 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:27.892215 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:27.892222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:27.893122 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:27.893623 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.893642 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.895050 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.895055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.896653 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.896666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.897937 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.897945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.898112 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.898815 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:27.898839 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:27.900117 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:27.904631 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.904638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.904645 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.905202 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.905660 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.905664 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.905668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.905835 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.906642 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:27.906650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:27.906894 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 02:20:27.907623 (+ 729us) client_negotiation.cc:240] Negotiation successful
0926 02:20:27.907975 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":465,"threads_started":1}
I20260926 02:20:27.910326 19938 leader_election.cc:302] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c8b2df03947a42809d10cbfcbbeb1bdb; no voters: 4926ed7f7fde4132b6b7a411a01bb9df
I20260926 02:20:27.911721 19993 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:20:27.911985 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.891581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.12.244.254:33863)
0926 02:20:27.891783 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:27.891789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:27.892864 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 02:20:27.893101 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:27.893108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:27.893220 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:27.893296 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:27.894802 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.895953 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.898009 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.899060 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.899189 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.901667 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:27.901672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:27.901675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:27.901710 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:27.904972 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.909470 (+ 4498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.909476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.909480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.909661 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.910157 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.910163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.910167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.910545 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:27.910677 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:27.911315 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 02:20:27.911608 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for oryx/127.12.244.252@KRBTEST.COM; defaulting to no policy
I20260926 02:20:27.913105 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.891995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:39002)
0926 02:20:27.892737 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:27.892763 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:27.892959 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:27.893524 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:27.893529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:27.894253 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:27.894588 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.894600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.896152 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.896157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.897843 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.897854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.899518 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.899525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.899714 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.900402 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:27.900418 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:27.901398 (+ 980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:27.908278 (+ 6880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.908285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.908289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.909277 (+ 988us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.909819 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.909826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.909835 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.910022 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.910779 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:27.910783 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:27.910998 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 02:20:27.912218 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0926 02:20:27.912597 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":1,"thread_start_us":395,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.252@KRBTEST.COM" created.
Sep 26 02:20:27 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389225, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.254@KRBTEST.COM for oryx/127.12.244.253@KRBTEST.COM
I20260926 02:20:27.944954 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.920625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.12.244.253:35731)
0926 02:20:27.920821 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:27.920827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:27.925362 (+ 4535us) server_negotiation.cc:407] Connection header received
0926 02:20:27.925832 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:27.925841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:27.925961 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:27.926090 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:27.927781 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.929079 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.931323 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.932195 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.932386 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.937360 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:27.937371 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:27.937378 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:27.937440 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:27.941185 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.942319 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.942337 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.942340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.942539 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:27.943040 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:27.943045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:27.943048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:27.943473 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:27.943645 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:27.944249 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 02:20:27.944557 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:20:27.945034 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:27.920596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.0.0.1:50590)
0926 02:20:27.925159 (+ 4563us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:27.925214 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:27.925540 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:27.926174 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:27.926182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:27.927186 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:27.927548 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.927564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.929279 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.929287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.931051 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:27.931072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:27.932446 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:27.932453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:27.932698 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:27.933608 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:27.933636 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:27.937013 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:27.941409 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.941429 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.941438 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.942160 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.942673 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:27.942678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:27.942682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:27.942877 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:27.943627 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:27.943637 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:27.943898 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 02:20:27.944124 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 02:20:27.944567 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4238,"thread_start_us":4056,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.252.keytab.
Entry for principal oryx/127.12.244.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.252.keytab.
I20260926 02:20:27.947974 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:32769
--webserver_interface=127.12.244.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.252.keytab
--principal=oryx/127.12.244.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:20:27.947911 19450 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.244.253:35731: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
member_type: NON_VOTER
last_known_addr {
host: "127.12.244.253"
port: 35731
}
attrs {
promote: true
}
}
dest_uuid: "4926ed7f7fde4132b6b7a411a01bb9df"
cas_config_opid_index: -1
I20260926 02:20:27.949254 19450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } }
I20260926 02:20:27.958735 19961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:27.957769 20016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:27.962064 19961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Refusing update from remote peer 4926ed7f7fde4132b6b7a411a01bb9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 02:20:27.963330 20017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:27.972049 20016 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.244.253:35731
I20260926 02:20:27.973089 19919 master.cc:537] Master@127.12.244.253:35731 shutting down...
I20260926 02:20:27.973083 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:27.974421 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:27.994066 19961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:28.019500 20027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:28.020085 20027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: LEARNER
I20260926 02:20:28.020045 20023 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:28.020392 20023 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: LEARNER
I20260926 02:20:28.020612 19993 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4926ed7f7fde4132b6b7a411a01bb9df. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:28.020876 19993 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: LEARNER
I20260926 02:20:28.023952 19919 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 LEARNER]: Raft consensus shutting down.
I20260926 02:20:28.024530 19919 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 LEARNER]: Raft consensus is shut down!
I20260926 02:20:28.024706 19919 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: stopping tablet replica
I20260926 02:20:28.033044 19919 master.cc:552] Master@127.12.244.253:35731 shutdown complete.
I20260926 02:20:28.033223 19919 master_runner.cc:305] Clearing existing system tablet
I20260926 02:20:28.033706 19919 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:28.033900 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:28.037211 19919 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:20:28.039196 20029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:28.039585 19919 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:28.040159 19919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 02:20:28.040305 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "122647edc149aeb2b9cb062792b4ce8e"
server_key_iv: "95cded08a38005e656506785dd7830f0"
server_key_version: "encryptionkey@0"
I20260926 02:20:28.045231 19919 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:28.053730 19919 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:20:28.054096 19919 log.cc:1196] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 02:20:28.054713 19919 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Deleting consensus metadata
I20260926 02:20:28.055075 19919 master_runner.cc:315] Copying system tablet from 127.12.244.254:33863
I20260926 02:20:28.058318 19919 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: tablet copy: Beginning tablet copy session from remote peer at address 127.12.244.254:33863
I20260926 02:20:28.077242 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.059669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.12.244.254:33863)
0926 02:20:28.059883 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.059890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.060144 (+ 254us) server_negotiation.cc:407] Connection header received
0926 02:20:28.060773 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.060779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.060894 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.061015 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.062263 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.063428 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.065548 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.066235 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.066385 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.069846 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.069854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.069860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.069907 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.073902 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.074884 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.074891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.074895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.075071 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.075545 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.075551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.075555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.075944 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.076096 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.076591 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.076852 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:20:28.077666 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.059372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:39006)
0926 02:20:28.059981 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.060055 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.060596 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.061063 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.061068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.061675 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.062058 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.062072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.063633 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.063641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.065357 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.065376 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.066543 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.066552 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.066813 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.067814 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.067848 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.069528 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.074114 (+ 4586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.074121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.074125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.074727 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.075229 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.075234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.075237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.075412 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.076100 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.076104 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.076297 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.076471 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.076938 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":538,"threads_started":1}
I20260926 02:20:28.078795 19471 tablet_copy_service.cc:140] P 4926ed7f7fde4132b6b7a411a01bb9df: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c8b2df03947a42809d10cbfcbbeb1bdb ({username='oryx', principal='oryx/127.12.244.253@KRBTEST.COM'} at 127.0.0.1:39006)
I20260926 02:20:28.079156 19471 tablet_copy_service.cc:161] P 4926ed7f7fde4132b6b7a411a01bb9df: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c8b2df03947a42809d10cbfcbbeb1bdb at {username='oryx', principal='oryx/127.12.244.253@KRBTEST.COM'} at 127.0.0.1:39006: session id = c8b2df03947a42809d10cbfcbbeb1bdb-00000000000000000000000000000000
I20260926 02:20:28.083886 19471 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:20:28.086685 19919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:20:28.094390 19919 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: tablet copy: Starting download of 0 data blocks...
I20260926 02:20:28.094771 19919 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: tablet copy: Starting download of 1 WAL segments...
I20260926 02:20:28.098002 19919 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:20:28.103207 19471 tablet_copy_service.cc:342] P 4926ed7f7fde4132b6b7a411a01bb9df: Request end of tablet copy session c8b2df03947a42809d10cbfcbbeb1bdb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.244.253@KRBTEST.COM'} at 127.0.0.1:39006
I20260926 02:20:28.103444 19471 tablet_copy_service.cc:433] P 4926ed7f7fde4132b6b7a411a01bb9df: ending tablet copy session c8b2df03947a42809d10cbfcbbeb1bdb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c8b2df03947a42809d10cbfcbbeb1bdb
I20260926 02:20:28.116983 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:28.120127 20036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:28.120260 20037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:28.122125 20039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:28.122740 19919 server_base.cc:1034] running on GCE node
I20260926 02:20:28.123214 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:28.123922 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:28.125177 19919 hybrid_clock.cc:648] HybridClock initialized: now 1790389228125101 us; error 34 us; skew 500 ppm
I20260926 02:20:28.125425 19919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389228, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:28.127777 19919 init.cc:377] Logged in from keytab as oryx/127.12.244.253@KRBTEST.COM (short username oryx)
I20260926 02:20:28.133718 19919 webserver.cc:460] Webserver started at http://127.12.244.253:45031/ using document root <none> and password file <none>
I20260926 02:20:28.134158 19919 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:28.134238 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:28.137601 19919 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 02:20:28.139703 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:28.140134 19919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 02:20:28.140327 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "122647edc149aeb2b9cb062792b4ce8e"
server_key_iv: "95cded08a38005e656506785dd7830f0"
server_key_version: "encryptionkey@0"
I20260926 02:20:28.140573 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:28.154114 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:28.158762 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:28.198930 19919 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.253:35731
I20260926 02:20:28.198962 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.253:35731 every 8 connection(s)
I20260926 02:20:28.200300 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
I20260926 02:20:28.208684 20098 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:20:28.213042 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Bootstrap starting.
I20260926 02:20:28.228466 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:20:28.229075 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: Bootstrap complete.
I20260926 02:20:28.230182 20098 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } }
I20260926 02:20:28.230639 20098 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8b2df03947a42809d10cbfcbbeb1bdb, State: Initialized, Role: LEARNER
I20260926 02:20:28.230968 20098 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } }
I20260926 02:20:28.232008 20101 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:28.232470 20101 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: LEARNER
I20260926 02:20:28.232506 20098 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:28.274952 20113 catalog_manager.cc:1259] Loaded cluster ID: f7663f0c98dd4b1c96e53242122e188c
I20260926 02:20:28.275174 20113 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: loading cluster ID for follower catalog manager: success
I20260926 02:20:28.279207 20113 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: acquiring CA information for follower catalog manager: success
I20260926 02:20:28.280184 19913 heartbeater.cc:503] Master 127.12.244.254:33863 was elected leader, sending a full tablet report...
I20260926 02:20:28.283600 20113 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:20:28.404860 20018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:28.405181 20018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:28.405366 20018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:28.413560 20018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:28.413726 20018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:28.413811 20018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:28.413913 20018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:28.413977 20018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:28.425554 20018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.252.keytab
--principal=oryx/127.12.244.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:32769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.244.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:28.427037 20018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:28.428529 20018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:28.441779 20118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:28.441910 20119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:28.441954 20121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:28.442553 20018 server_base.cc:1034] running on GCE node
I20260926 02:20:28.443600 20018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:28.445313 20018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:28.446576 20018 hybrid_clock.cc:648] HybridClock initialized: now 1790389228446529 us; error 43 us; skew 500 ppm
I20260926 02:20:28.446985 20018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389228, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:28.450713 20018 init.cc:377] Logged in from keytab as oryx/127.12.244.252@KRBTEST.COM (short username oryx)
I20260926 02:20:28.452615 20018 webserver.cc:460] Webserver started at http://127.12.244.252:43377/ using document root <none> and password file <none>
I20260926 02:20:28.453403 20018 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:28.453558 20018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:28.453908 20018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:28.457356 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/instance:
uuid: "14b6519af9cc47d7a99b11fab42854b1"
format_stamp: "Formatted at 2026-09-26 02:20:28 on dist-test-slave-qfmw"
server_key: "169f938419bfb558ba40185873a4b6b1"
server_key_iv: "868913b4a654f7834a8293f09f6604ae"
server_key_version: "encryptionkey@0"
I20260926 02:20:28.458217 20018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal/instance:
uuid: "14b6519af9cc47d7a99b11fab42854b1"
format_stamp: "Formatted at 2026-09-26 02:20:28 on dist-test-slave-qfmw"
server_key: "169f938419bfb558ba40185873a4b6b1"
server_key_iv: "868913b4a654f7834a8293f09f6604ae"
server_key_version: "encryptionkey@0"
I20260926 02:20:28.464308 20018 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:20:28.468729 20128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:28.470391 20018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 02:20:28.470628 20018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
uuid: "14b6519af9cc47d7a99b11fab42854b1"
format_stamp: "Formatted at 2026-09-26 02:20:28 on dist-test-slave-qfmw"
server_key: "169f938419bfb558ba40185873a4b6b1"
server_key_iv: "868913b4a654f7834a8293f09f6604ae"
server_key_version: "encryptionkey@0"
I20260926 02:20:28.470923 20018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:28.489482 20018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:28.494822 20018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:28.495245 20018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:28.518133 20018 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.252:32769
I20260926 02:20:28.518134 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.252:32769 every 8 connection(s)
I20260926 02:20:28.520264 20018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
I20260926 02:20:28.521149 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:28.521546 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:28.526080 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 20018
I20260926 02:20:28.553536 20181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:28.558820 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.522354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.12.244.253:35731)
0926 02:20:28.551561 (+ 29207us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.551571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.551596 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:20:28.551672 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.551678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.551795 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.552504 (+ 709us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:28.552709 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.554433 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.556690 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.557629 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.557865 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.558066 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.558312 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.558459 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29050,"thread_start_us":30614,"threads_started":1}
I20260926 02:20:28.558849 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.522139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.0.0.1:50606)
0926 02:20:28.546448 (+ 24309us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.546483 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.546719 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.552052 (+ 5333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.552058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.552093 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:28.552324 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.552336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.554651 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.554659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.556504 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.556520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.557842 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.557850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.558067 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.558144 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.558285 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.558445 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24076,"thread_start_us":24712,"threads_started":1}
I20260926 02:20:28.560730 20101 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4926ed7f7fde4132b6b7a411a01bb9df. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: NON_VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: true } } }
I20260926 02:20:28.561014 20101 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: LEARNER
I20260926 02:20:28.561656 20182 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: attempting to promote NON_VOTER c8b2df03947a42809d10cbfcbbeb1bdb to VOTER
I20260926 02:20:28.562602 20182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } }
I20260926 02:20:28.564815 20073 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 LEARNER]: Refusing update from remote peer 4926ed7f7fde4132b6b7a411a01bb9df: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 02:20:28.565500 20180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.252@KRBTEST.COM
I20260926 02:20:28.566435 20181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:20:28.569609 20182 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } }
I20260926 02:20:28.572821 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } }
I20260926 02:20:28.573113 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:28.573740 20180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } }
I20260926 02:20:28.573983 20180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:28.573902 20073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } }
I20260926 02:20:28.577816 20101 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } }
I20260926 02:20:28.578059 20101 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:20:28.578505 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.529756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.12.244.252:32769)
0926 02:20:28.550166 (+ 20410us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.550187 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.550253 (+ 66us) server_negotiation.cc:407] Connection header received
0926 02:20:28.552645 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.552676 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.553221 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.553693 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.555823 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.557801 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.560207 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.561343 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.561616 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.568093 (+ 6477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.568109 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.568135 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.568202 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.572837 (+ 4635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.573927 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.573936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.573948 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.574215 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.574750 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.574756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.574759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.575527 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.575885 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.576401 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.576920 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20115,"thread_start_us":19815,"threads_started":1}
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389228, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.252@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:28.589401 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.573034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.12.244.254:33863)
0926 02:20:28.573236 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.573241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.573260 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:20:28.573322 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.573326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.573407 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.573511 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.575290 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.576491 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.578993 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.579516 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.579627 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.583644 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.583650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.583653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.583689 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.586657 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.587618 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.587622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.587624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.587762 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.588116 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.588119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.588121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.588406 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.588515 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.588980 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.589144 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:20:28.589442 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.570342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:39014)
0926 02:20:28.571285 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.571340 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.571502 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.573599 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.573609 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.574587 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.575028 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.575041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.576930 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.576936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.578830 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.578847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.579722 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.579729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.579934 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.580751 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.580770 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.583415 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.586844 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.586857 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.586864 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.587496 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.587865 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.587869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.587871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.588008 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.588515 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.588523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.588688 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.588856 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.589126 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":506,"threads_started":1}
I20260926 02:20:28.590899 20181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389228, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.252@KRBTEST.COM for oryx/127.12.244.253@KRBTEST.COM
I20260926 02:20:28.604967 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.591754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.0.0.1:50620)
0926 02:20:28.591989 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.592005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.592092 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.592413 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.592417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.592747 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.592888 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.592895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.594174 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.594178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.595260 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.595272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.596060 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.596065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.596223 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.596817 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.596842 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.598977 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.602370 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.602375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.602378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.602801 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.603342 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.603358 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.603360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.603495 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.604110 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.604115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.604303 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.604444 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.604705 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:20:28.605185 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.591819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.12.244.253:35731)
0926 02:20:28.592077 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.592082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.592099 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:20:28.592188 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.592192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.592272 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.592357 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.593039 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.594046 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.595403 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.595878 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.595987 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.599246 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.599252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.599256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.599292 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.602195 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.602973 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.602977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.602979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.603230 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.603625 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.603630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.603633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.603962 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.604086 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.604579 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.604860 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:20:28.606385 20181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389228, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.252@KRBTEST.COM for oryx/127.12.244.252@KRBTEST.COM
I20260926 02:20:28.620075 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.607280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.0.0.1:44046)
0926 02:20:28.607460 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.607481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.607598 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.607985 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.607989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.608369 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.608513 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.608521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.610000 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.610004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.610880 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.610891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.611735 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.611739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.611895 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.612504 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.612521 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.614615 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.617827 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.617832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.617835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.618202 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.618532 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.618535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.618538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.618680 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.619288 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.619291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.619439 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.619550 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.619760 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 02:20:28.620378 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.607535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.12.244.252:32769)
0926 02:20:28.607696 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.607700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.607716 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:20:28.607763 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.607767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.607829 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.607917 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.608668 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.609857 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.611028 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.611558 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.611667 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.614898 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.614902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.614906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.614937 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.617676 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.618313 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.618316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.618319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.618440 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.618821 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.618824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.618826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.619174 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.619295 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.619644 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.619959 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":57}
I20260926 02:20:28.626494 20181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Bootstrap starting.
I20260926 02:20:28.629881 20181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:28.631016 20181 log.cc:824] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:28.634513 20181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: No bootstrap required, opened a new log
I20260926 02:20:28.642809 20181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } }
I20260926 02:20:28.643286 20181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:28.643399 20181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6519af9cc47d7a99b11fab42854b1, State: Initialized, Role: FOLLOWER
I20260926 02:20:28.644006 20181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } }
I20260926 02:20:28.644963 20192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } } }
I20260926 02:20:28.645375 20192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:20:28.646090 20181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 02:20:28.656134 20203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:20:28.656340 20203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:20:28.660293 20018 master_runner.cc:428] Detected that this master 14b6519af9cc47d7a99b11fab42854b1 is joining an existing cluster
I20260926 02:20:28.660449 20018 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.244.252:32769
I20260926 02:20:28.685612 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 19513
I20260926 02:20:28.689059 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.665451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.12.244.252:32769)
0926 02:20:28.665663 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.665670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.671559 (+ 5889us) server_negotiation.cc:407] Connection header received
0926 02:20:28.671620 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.671625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.671708 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.671807 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.673618 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.674807 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.677066 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.677894 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.678026 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.680989 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.680995 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.681000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.681042 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.685048 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.685948 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.685954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.685957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.686109 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.686558 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.686563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.686566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.686957 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.687163 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.687676 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.687910 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":58}
I20260926 02:20:28.689688 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.665451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.0.0.1:44054)
0926 02:20:28.671321 (+ 5870us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.671355 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.671458 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.671889 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.671896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.672991 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.673435 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.673449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.675006 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.675011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.676890 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.676906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.678446 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.678454 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.678651 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.679440 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.679469 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.680791 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.685208 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.685213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.685218 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.685778 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.686272 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.686277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.686280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.686437 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.687239 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.687244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.687416 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.688775 (+ 1359us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.689103 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5744,"mutex_wait_us":3,"thread_start_us":5215,"threads_started":1}
I20260926 02:20:28.690194 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.665822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.12.244.253:35731)
0926 02:20:28.666061 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.666071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.670848 (+ 4777us) server_negotiation.cc:407] Connection header received
0926 02:20:28.671229 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.671235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.671321 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.671408 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.673887 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.675002 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.679212 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.679749 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.679838 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.682777 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.682784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.682789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.682832 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.686292 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.687184 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.687190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.687194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.687377 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.687912 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.687920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.687926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.688267 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.688425 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.689432 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.689676 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:20:28.691218 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.665356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.0.0.1:50630)
0926 02:20:28.670679 (+ 5323us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.670708 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.670999 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.672022 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.672026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.673032 (+ 1006us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.673667 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.673680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.675283 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.675291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.677894 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.677903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.680073 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.680078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.680213 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.680844 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.680869 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.682530 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.686466 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.686471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.686474 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.686985 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.687532 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.687537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.687540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.687774 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.688503 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.688507 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.688893 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.690367 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.690708 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5076,"mutex_wait_us":397,"thread_start_us":4827,"threads_started":1}
I20260926 02:20:28.706959 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.665120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39022 (local address 127.12.244.254:33863)
0926 02:20:28.665334 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.665341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.671973 (+ 6632us) server_negotiation.cc:407] Connection header received
0926 02:20:28.672367 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.672373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.672481 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.672562 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.674071 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.675198 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.678055 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.678657 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.678736 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.698957 (+ 20221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.698968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.698974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.699040 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.702856 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.703877 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.703885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.703888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.704063 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.704650 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.704657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.704660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.705035 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.705207 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.706325 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.706560 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:20:28.707062 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.666603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:39022)
0926 02:20:28.671875 (+ 5272us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.671891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.672163 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.672778 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.672785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.673514 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.673893 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.673904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.675387 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.675394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.677529 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.677545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.679023 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.679034 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.680664 (+ 1630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.681388 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.681410 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.698363 (+ 16953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.703086 (+ 4723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.703124 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.703129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.703705 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.704237 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.704244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.704250 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.704491 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.705404 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.705408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.706016 (+ 608us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.706180 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.706557 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5140,"mutex_wait_us":64,"spinlock_wait_cycles":33664,"thread_start_us":4398,"threads_started":1}
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.193@KRBTEST.COM
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389225, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.254@KRBTEST.COM for oryx/127.12.244.252@KRBTEST.COM
I20260926 02:20:28.724663 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.687496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.12.244.193:45503)
0926 02:20:28.700601 (+ 13105us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.700612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.700647 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:20:28.700727 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.700736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.701061 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.701315 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.705245 (+ 3930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.706921 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.709356 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.710353 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.710581 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.716027 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.716052 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.716075 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.716528 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.720907 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.721934 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.721943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.721945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.722116 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.722646 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.722653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.722658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.723186 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.723516 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.723916 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.724212 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11979,"thread_start_us":12249,"threads_started":1}
I20260926 02:20:28.726280 19638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:28.730028 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.712690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.0.0.1:44070)
0926 02:20:28.712858 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.712879 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.712978 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.713471 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.713477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.713975 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.714150 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.714162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.716224 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.716230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.717133 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.717146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.718121 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.718126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.718316 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.718923 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.718950 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.721132 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.725527 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.725534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.725538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.726124 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.726626 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.726634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.726641 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.726872 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.728309 (+ 1437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.728316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.728576 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.728888 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.729438 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 02:20:28.731187 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.712801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.12.244.252:32769)
0926 02:20:28.712994 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.713000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.713021 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:20:28.713132 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.713138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.713284 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.713419 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.714928 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.716085 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.717314 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.717959 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.718106 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.721355 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.721364 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.721369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.721418 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.725361 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.726269 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.726277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.726280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.726444 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.727068 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.727075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.727082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.728131 (+ 1049us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.730351 (+ 2220us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.730522 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.730784 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:20:28.733567 19450 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.244.252:32769: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1"
member_type: NON_VOTER
last_known_addr {
host: "127.12.244.252"
port: 32769
}
attrs {
promote: true
}
}
dest_uuid: "4926ed7f7fde4132b6b7a411a01bb9df"
cas_config_opid_index: 6
I20260926 02:20:28.734527 19450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } }
I20260926 02:20:28.737017 20073 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Refusing update from remote peer 4926ed7f7fde4132b6b7a411a01bb9df: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 02:20:28.737736 20019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 02:20:28.738147 20155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:28.741302 20180 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:28.742020 20073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:28.742612 20155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Refusing update from remote peer 4926ed7f7fde4132b6b7a411a01bb9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 02:20:28.743695 20182 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:28.745147 20180 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.244.252:32769
I20260926 02:20:28.746870 20018 master.cc:537] Master@127.12.244.252:32769 shutting down...
I20260926 02:20:28.747334 20101 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:28.747649 20101 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: FOLLOWER
W20260926 02:20:28.748149 20134 tablet.cc:2369] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:20:28.749389 20019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:28.749761 20019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:28.749974 20192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260926 02:20:28.749572 19437 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 02:20:28.750207 20192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } }
I20260926 02:20:28.750084 20214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:28.750999 20214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:28.752671 20192 leader_election.cc:288] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863), c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253:35731)
I20260926 02:20:28.753091 19461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14b6519af9cc47d7a99b11fab42854b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" is_pre_election: true
I20260926 02:20:28.753306 20073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14b6519af9cc47d7a99b11fab42854b1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" is_pre_election: true
W20260926 02:20:28.753480 19437 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df -> Peer 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252:32769): Couldn't send request to peer 14b6519af9cc47d7a99b11fab42854b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 02:20:28.754226 20129 leader_election.cc:302] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14b6519af9cc47d7a99b11fab42854b1; no voters: 4926ed7f7fde4132b6b7a411a01bb9df, c8b2df03947a42809d10cbfcbbeb1bdb
I20260926 02:20:28.754946 20192 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 02:20:28.761619 20018 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:28.762079 20018 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:28.762290 20018 tablet_replica.cc:330] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: stopping tablet replica
I20260926 02:20:28.769294 20018 master.cc:552] Master@127.12.244.252:32769 shutdown complete.
I20260926 02:20:28.769392 20018 master_runner.cc:305] Clearing existing system tablet
I20260926 02:20:28.769795 20018 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:28.769908 20018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:28.772974 20018 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260926 02:20:28.774731 20217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:28.775166 20018 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:28.775524 20018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 02:20:28.775655 20018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
uuid: "14b6519af9cc47d7a99b11fab42854b1"
format_stamp: "Formatted at 2026-09-26 02:20:28 on dist-test-slave-qfmw"
server_key: "169f938419bfb558ba40185873a4b6b1"
server_key_iv: "868913b4a654f7834a8293f09f6604ae"
server_key_version: "encryptionkey@0"
I20260926 02:20:28.780866 20018 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:28.788340 20018 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 02:20:28.788601 20018 log.cc:1196] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 02:20:28.789137 20018 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Deleting consensus metadata
I20260926 02:20:28.789430 20018 master_runner.cc:315] Copying system tablet from 127.12.244.254:33863
I20260926 02:20:28.792136 20018 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: tablet copy: Beginning tablet copy session from remote peer at address 127.12.244.254:33863
I20260926 02:20:28.810971 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.793013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:39028)
0926 02:20:28.795569 (+ 2556us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:28.795597 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:28.795789 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:28.796209 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:28.796215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:28.796766 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:28.797018 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.797030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.798468 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.798474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.800050 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:28.800067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.801097 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.801103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:28.801295 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.802517 (+ 1222us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:28.802544 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:28.803746 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:28.807899 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.807908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.807912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.808464 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.808949 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:28.808955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:28.808959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:28.809144 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:28.809794 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:28.809799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:28.809985 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 02:20:28.810129 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 02:20:28.810490 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2385,"thread_start_us":2244,"threads_started":1}
I20260926 02:20:28.812039 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:28.793086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.12.244.254:33863)
0926 02:20:28.793282 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:28.793288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:28.795732 (+ 2444us) server_negotiation.cc:407] Connection header received
0926 02:20:28.795936 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:28.795943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:28.796041 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:28.796155 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:28.797205 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.798298 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.800235 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:28.800922 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:28.801062 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:28.803987 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:28.803995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:28.804000 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:28.804042 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:28.807719 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.808627 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.808632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.808636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.808806 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:28.809293 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:28.809299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:28.809303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:28.809664 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:28.809817 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:28.811418 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0926 02:20:28.811673 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:20:28.812608 19471 tablet_copy_service.cc:140] P 4926ed7f7fde4132b6b7a411a01bb9df: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 14b6519af9cc47d7a99b11fab42854b1 ({username='oryx', principal='oryx/127.12.244.252@KRBTEST.COM'} at 127.0.0.1:39028)
I20260926 02:20:28.812783 19471 tablet_copy_service.cc:161] P 4926ed7f7fde4132b6b7a411a01bb9df: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 14b6519af9cc47d7a99b11fab42854b1 at {username='oryx', principal='oryx/127.12.244.252@KRBTEST.COM'} at 127.0.0.1:39028: session id = 14b6519af9cc47d7a99b11fab42854b1-00000000000000000000000000000000
I20260926 02:20:28.816449 19471 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 02:20:28.818569 20018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 02:20:28.826498 20018 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: tablet copy: Starting download of 0 data blocks...
I20260926 02:20:28.826861 20018 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: tablet copy: Starting download of 1 WAL segments...
I20260926 02:20:28.829540 20018 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 02:20:28.834515 19471 tablet_copy_service.cc:342] P 4926ed7f7fde4132b6b7a411a01bb9df: Request end of tablet copy session 14b6519af9cc47d7a99b11fab42854b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.244.252@KRBTEST.COM'} at 127.0.0.1:39028
I20260926 02:20:28.834672 19471 tablet_copy_service.cc:433] P 4926ed7f7fde4132b6b7a411a01bb9df: ending tablet copy session 14b6519af9cc47d7a99b11fab42854b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 14b6519af9cc47d7a99b11fab42854b1
I20260926 02:20:28.841835 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 19513
I20260926 02:20:28.850375 20018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:28.857062 20224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:28.865705 20225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:28.871352 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:45503
--local_ip_for_outbound_sockets=127.12.244.193
--tserver_master_addrs=127.12.244.254:33863
--webserver_port=41767
--webserver_interface=127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab
--principal=oryx/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:28.880612 20227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:28.882436 20018 server_base.cc:1034] running on GCE node
I20260926 02:20:28.883033 20018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:28.883924 20018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:28.885164 20018 hybrid_clock.cc:648] HybridClock initialized: now 1790389228885111 us; error 39 us; skew 500 ppm
I20260926 02:20:28.885491 20018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:28 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389228, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:28.888218 20018 init.cc:377] Logged in from keytab as oryx/127.12.244.252@KRBTEST.COM (short username oryx)
I20260926 02:20:28.893096 20018 webserver.cc:460] Webserver started at http://127.12.244.252:41089/ using document root <none> and password file <none>
I20260926 02:20:28.893535 20018 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:28.893654 20018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:28.897512 20018 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:28.903514 20235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:28.905483 20018 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.002s
I20260926 02:20:28.905710 20018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
uuid: "14b6519af9cc47d7a99b11fab42854b1"
format_stamp: "Formatted at 2026-09-26 02:20:28 on dist-test-slave-qfmw"
server_key: "169f938419bfb558ba40185873a4b6b1"
server_key_iv: "868913b4a654f7834a8293f09f6604ae"
server_key_version: "encryptionkey@0"
I20260926 02:20:28.905974 20018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:28.951817 20018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:28.957135 20018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:29.008308 20018 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.252:32769
I20260926 02:20:29.008333 20287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.252:32769 every 8 connection(s)
I20260926 02:20:29.009388 20018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
I20260926 02:20:29.017014 20288 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:20:29.019932 20288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Bootstrap starting.
I20260926 02:20:29.036711 20288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:20:29.037246 20288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Bootstrap complete.
I20260926 02:20:29.038208 20288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } }
I20260926 02:20:29.038506 20288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14b6519af9cc47d7a99b11fab42854b1, State: Initialized, Role: LEARNER
I20260926 02:20:29.038753 20288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } }
I20260926 02:20:29.039606 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:29.040032 20288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:29.040086 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:20:29.055260 20302 catalog_manager.cc:1259] Loaded cluster ID: f7663f0c98dd4b1c96e53242122e188c
I20260926 02:20:29.055392 20302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: loading cluster ID for follower catalog manager: success
I20260926 02:20:29.058316 20302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: acquiring CA information for follower catalog manager: success
I20260926 02:20:29.060616 20302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:20:29.335963 20229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:29.336278 20229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:29.336373 20229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:29.345708 20229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:29.345840 20229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:29.346019 20229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:29.348588 20306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.341903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.0.0.1:44078)
0926 02:20:29.342622 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:29.342648 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:29.342817 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:29.343361 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:29.343366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:29.343395 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:29.343652 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.343662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.345032 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.345040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.346471 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.346489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.347750 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.347757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.347939 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.347975 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:20:29.348094 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 02:20:29.348220 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":465,"threads_started":1}
I20260926 02:20:29.348721 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.342097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.12.244.252:32769)
0926 02:20:29.342939 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:29.342946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:29.342963 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:20:29.343035 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:29.343038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:29.343193 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:29.343342 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:29.343818 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.344888 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.346621 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.347566 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.347745 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.347957 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:29.348160 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 02:20:29.348323 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":545,"threads_started":1}
I20260926 02:20:29.349915 20263 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260926 02:20:29.350456 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4926ed7f7fde4132b6b7a411a01bb9df. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:29.350729 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: This master's current role is: LEARNER
I20260926 02:20:29.357133 20229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab
--principal=oryx/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:45503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=41767
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:29.358613 20229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:29.360148 20229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:29.374742 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:29.374692 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:29.374781 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:29.376315 20229 server_base.cc:1034] running on GCE node
I20260926 02:20:29.376982 20229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:29.378006 20229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:29.379211 20229 hybrid_clock.cc:648] HybridClock initialized: now 1790389229379117 us; error 86 us; skew 500 ppm
I20260926 02:20:29.379567 20229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:29 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389229, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:29.383793 20229 init.cc:377] Logged in from keytab as oryx/127.12.244.193@KRBTEST.COM (short username oryx)
I20260926 02:20:29.385494 20229 webserver.cc:460] Webserver started at http://127.12.244.193:41767/ using document root <none> and password file <none>
I20260926 02:20:29.386360 20229 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:29.386523 20229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:29.393131 20229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 02:20:29.396466 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:29.398469 20229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:20:29.398698 20229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "a5ee8b1e6a49c09f90ef66714e663026"
server_key_iv: "312b145cdca954b50ae4355336e8f898"
server_key_version: "encryptionkey@0"
I20260926 02:20:29.399883 20229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:29.416391 20229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:29.421746 20229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:29.422147 20229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:29.423393 20229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:29.425531 20229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:29.425669 20229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:29.425807 20229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:29.425899 20229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:29.466727 20229 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45503
I20260926 02:20:29.466786 20432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45503 every 8 connection(s)
I20260926 02:20:29.468327 20229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:29.475201 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 20229
I20260926 02:20:29.475564 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 19649
Sep 26 02:20:29 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389229, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.252@KRBTEST.COM
I20260926 02:20:29.506390 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.473775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:60693 (local address 127.12.244.252:32769)
0926 02:20:29.474040 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:29.474050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:29.480286 (+ 6236us) server_negotiation.cc:407] Connection header received
0926 02:20:29.485316 (+ 5030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:29.485321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:29.485391 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:29.485509 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:29.487393 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.488479 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.491319 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.492461 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.492583 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.497938 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:29.497947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:29.497952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:29.497999 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:29.501680 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.502940 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.502946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.502950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.503137 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.503864 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.503869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.503873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.504241 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:29.504380 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:29.505625 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0926 02:20:29.505914 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 02:20:29 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389229, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.253@KRBTEST.COM
I20260926 02:20:29.509639 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.472898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.12.244.193:60693)
0926 02:20:29.479995 (+ 7097us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:29.480085 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:29.483926 (+ 3841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:29.485810 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:29.485820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:29.486437 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:29.487075 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.487170 (+ 95us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.488693 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.488698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.491136 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.491154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.492980 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.492988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.493308 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.494266 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:29.494306 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:29.497586 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:29.501881 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:29.501894 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:29.501919 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:29.502774 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:29.503313 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:29.503318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:29.503322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:29.503721 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:29.504476 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:29.504484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:29.505147 (+ 663us) client_negotiation.cc:769] Sending connection context
0926 02:20:29.506935 (+ 1788us) client_negotiation.cc:240] Negotiation successful
0926 02:20:29.507711 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6609,"thread_start_us":6323,"threads_started":1}
Sep 26 02:20:29 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389229, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:29.511911 20433 heartbeater.cc:347] Connected to a master server at 127.12.244.252:32769
I20260926 02:20:29.512250 20433 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:29.513684 20433 heartbeater.cc:511] Master 127.12.244.252:32769 requested a full tablet report, sending...
I20260926 02:20:29.516606 20253 ts_manager.cc:194] Registered new tserver with Master: f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193:45503)
I20260926 02:20:29.518837 20441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.474111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.12.244.193:54917)
0926 02:20:29.500051 (+ 25940us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:29.500070 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:29.500241 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:29.500877 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:29.500884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:29.501538 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:29.501831 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.501843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.503327 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.503332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.504925 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.504939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.506155 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.506162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.506355 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.507121 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:29.507149 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:29.511715 (+ 4566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:29.515904 (+ 4189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:29.515910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:29.515914 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:29.516356 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:29.516869 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:29.516872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:29.516874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:29.516997 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:29.517732 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:29.517737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:29.517910 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 02:20:29.518056 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 02:20:29.518389 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25828,"thread_start_us":26225,"threads_started":1}
I20260926 02:20:29.519834 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.471970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:54917 (local address 127.12.244.253:35731)
0926 02:20:29.499434 (+ 27464us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:29.499441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:29.500167 (+ 726us) server_negotiation.cc:407] Connection header received
0926 02:20:29.500369 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:29.500375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:29.500472 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:29.500577 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:29.502021 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.503164 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.505097 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.505742 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.505867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.511962 (+ 6095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:29.511969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:29.511973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:29.512018 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:29.515750 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.516507 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.516512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.516514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.516634 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.517127 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.517132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.517134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.517517 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:29.517669 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:29.519246 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0926 02:20:29.519476 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27314,"thread_start_us":27154,"threads_started":1}
Sep 26 02:20:29 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.194@KRBTEST.COM
I20260926 02:20:29.520684 20435 heartbeater.cc:347] Connected to a master server at 127.12.244.253:35731
I20260926 02:20:29.520860 20435 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:29.521485 20435 heartbeater.cc:511] Master 127.12.244.253:35731 requested a full tablet report, sending...
I20260926 02:20:29.521651 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.472898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:35487)
0926 02:20:29.484894 (+ 11996us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:29.484912 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:29.485018 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:29.501910 (+ 16892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:29.501915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:29.502526 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:29.502870 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.502883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.505069 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.505079 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.506675 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:29.506692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.507851 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.507858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:29.508069 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.508732 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:29.508754 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:29.511351 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:29.518055 (+ 6704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:29.518060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:29.518065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:29.518590 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:29.519618 (+ 1028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:29.519629 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:29.519636 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:29.519833 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:29.520503 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:29.520511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:29.520678 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 02:20:29.520830 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 02:20:29.521176 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11796,"mutex_wait_us":42,"spinlock_wait_cycles":65408,"thread_start_us":11432,"threads_started":1}
I20260926 02:20:29.522836 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.473668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:35487 (local address 127.12.244.254:33863)
0926 02:20:29.500905 (+ 27237us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:29.500911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:29.500934 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:20:29.500996 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:29.501004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:29.501084 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:29.501206 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:29.503556 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.504908 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.506899 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.507618 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.508414 (+ 796us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.511603 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:29.511610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:29.511615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:29.511656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:29.517883 (+ 6227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.518748 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.518754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.518765 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.518933 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.519994 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.520000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.520005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.520355 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:29.522032 (+ 1677us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:29.522163 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 02:20:29.522413 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27098,"thread_start_us":27712,"threads_started":1}
I20260926 02:20:29.523391 20063 ts_manager.cc:194] Registered new tserver with Master: f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193:45503)
I20260926 02:20:29.524003 20434 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33863
I20260926 02:20:29.524168 20434 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:29.524682 20434 heartbeater.cc:511] Master 127.12.244.254:33863 requested a full tablet report, sending...
I20260926 02:20:29.526003 19450 ts_manager.cc:194] Re-registered known tserver with Master: f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193:45503)
I20260926 02:20:29.527328 19450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:35487
I20260926 02:20:29.530498 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:29.477109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.12.244.194:39225)
0926 02:20:29.487681 (+ 10572us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:29.487690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:29.488435 (+ 745us) server_negotiation.cc:407] Connection header received
0926 02:20:29.488513 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:29.488525 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:29.488832 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:29.488989 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:29.491319 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.493287 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.495184 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:29.496132 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:29.496302 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:29.521994 (+ 25692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:29.522016 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:29.522038 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:29.522110 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:29.526854 (+ 4744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.528239 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.528248 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.528250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.528393 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:29.528791 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:29.528797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:29.528800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:29.529158 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:29.529482 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:29.529797 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 02:20:29.530084 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10386,"thread_start_us":10216,"threads_started":1}
I20260926 02:20:29.531889 19773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:29.618117 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 19649
I20260926 02:20:29.635882 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:39225
--local_ip_for_outbound_sockets=127.12.244.194
--tserver_master_addrs=127.12.244.254:33863
--webserver_port=46043
--webserver_interface=127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab
--principal=oryx/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:20:29.813036 20214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: NON_VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: true } } }
I20260926 02:20:29.813376 20214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:29.814850 20446 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: attempting to promote NON_VOTER 14b6519af9cc47d7a99b11fab42854b1 to VOTER
I20260926 02:20:29.815613 20446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:29.817950 20073 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Refusing update from remote peer 4926ed7f7fde4132b6b7a411a01bb9df: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:20:29.818055 20263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 LEARNER]: Refusing update from remote peer 4926ed7f7fde4132b6b7a411a01bb9df: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 02:20:29.818645 20214 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:20:29.819062 20446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 02:20:29.822202 20214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.823011 20073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.825140 20214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.825337 20214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:29.825613 20214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.825747 20214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:29.827384 20452 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.828028 20452 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:20:29.828342 20447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.828658 20447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:29.829870 20263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.834467 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:29.834748 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 02:20:29.992848 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:29.993157 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:29.993328 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:30.000890 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:30.001015 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:30.001202 20445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:30.016371 20445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab
--principal=oryx/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:39225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=46043
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:30.018172 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:30.019956 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:30.035077 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:30.035383 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:30.035068 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:30.036126 20445 server_base.cc:1034] running on GCE node
I20260926 02:20:30.036935 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:30.038023 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:30.039254 20445 hybrid_clock.cc:648] HybridClock initialized: now 1790389230039200 us; error 54 us; skew 500 ppm
I20260926 02:20:30.039657 20445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:30.043334 20445 init.cc:377] Logged in from keytab as oryx/127.12.244.194@KRBTEST.COM (short username oryx)
I20260926 02:20:30.045058 20445 webserver.cc:460] Webserver started at http://127.12.244.194:46043/ using document root <none> and password file <none>
I20260926 02:20:30.045817 20445 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:30.045989 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:30.052616 20445 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:30.056177 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:30.058045 20445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:20:30.058280 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "f6a302db476745678497bc364527197a"
format_stamp: "Formatted at 2026-09-26 02:20:26 on dist-test-slave-qfmw"
server_key: "8d71e0ec492c71734eacbb6e1fefcab6"
server_key_iv: "35c357efc7cf504c9b647cced3f7a92f"
server_key_version: "encryptionkey@0"
I20260926 02:20:30.059470 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:30.079286 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:30.084182 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:30.084482 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:30.085796 20445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:30.087909 20445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:30.088049 20445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:30.088212 20445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:30.088316 20445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:30.126616 20445 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:39225
I20260926 02:20:30.126662 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:39225 every 8 connection(s)
I20260926 02:20:30.128360 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:30.129051 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 20445
I20260926 02:20:30.129397 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 19784
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.252@KRBTEST.COM
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.253@KRBTEST.COM
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.195@KRBTEST.COM
I20260926 02:20:30.192941 20588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.132152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:47811 (local address 127.12.244.254:33863)
0926 02:20:30.158059 (+ 25907us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.158068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.158100 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:20:30.162579 (+ 4479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.162590 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.162722 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.162832 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.171685 (+ 8853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.173059 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.176632 (+ 3573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.177384 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.177503 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.183819 (+ 6316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.183827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.183833 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.183893 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.188434 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.190261 (+ 1827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.190270 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.190274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.190448 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.190927 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.190934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.190940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.191392 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.191548 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.192242 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.192542 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25742,"thread_start_us":25940,"threads_started":1}
I20260926 02:20:30.192961 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.137280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:39429 (local address 127.12.244.252:32769)
0926 02:20:30.164943 (+ 27663us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.164955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.167066 (+ 2111us) server_negotiation.cc:407] Connection header received
0926 02:20:30.167182 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.167189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.167281 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.167414 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.171685 (+ 4271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.173059 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.176633 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.177596 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.177706 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.184093 (+ 6387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.184103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.184109 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.184158 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.187877 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.189026 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.189032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.189036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.189263 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.189961 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.189967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.189972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.190436 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.190649 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.192253 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.192535 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27450,"thread_start_us":27687,"threads_started":1}
I20260926 02:20:30.194842 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.137756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.12.244.194:44497)
0926 02:20:30.167197 (+ 29441us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.167225 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.167413 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.168172 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.168177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.168957 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:30.171443 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.171459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.173130 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.173138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.176394 (+ 3256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.176416 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.177952 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.177961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.178805 (+ 844us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.179846 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:30.179871 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:30.184100 (+ 4229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:30.189385 (+ 5285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.189391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.189396 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.190434 (+ 1038us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.190973 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.190980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.190986 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.191185 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.192099 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:30.192108 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:30.192255 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.192442 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.193074 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29245,"mutex_wait_us":430,"spinlock_wait_cycles":768,"thread_start_us":15194,"threads_started":1}
I20260926 02:20:30.195279 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.136700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.12.244.194:39429)
0926 02:20:30.166663 (+ 29963us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.166693 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.166899 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.167639 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.167644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.168370 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:30.171418 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.171454 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.173254 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.173259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.176416 (+ 3157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.176434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.178028 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.178034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.178491 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.179596 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:30.179636 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:30.183787 (+ 4151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:30.188120 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.188133 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.188155 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.188871 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.189388 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.189393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.189396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.189716 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.190754 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:30.190769 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:30.191720 (+ 951us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.193796 (+ 2076us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.194104 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29447,"mutex_wait_us":711,"spinlock_wait_cycles":8320,"thread_start_us":29531,"threads_started":1}
I20260926 02:20:30.194864 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.133115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:47811)
0926 02:20:30.153628 (+ 20513us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.153715 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.161280 (+ 7565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.163636 (+ 2356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.163654 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.167842 (+ 4188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:30.171417 (+ 3575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.171454 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.173268 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.173272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.176416 (+ 3144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.176434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.177930 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.177940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.178497 (+ 557us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.179638 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:30.179659 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:30.183397 (+ 3738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:30.189501 (+ 6104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.189507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.189510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.190070 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.190594 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.190599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.190602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.190776 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.191634 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:30.191639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:30.191814 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.193154 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.193470 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20007,"mutex_wait_us":492,"spinlock_wait_cycles":52736,"thread_start_us":33508,"threads_started":1}
I20260926 02:20:30.195643 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.138420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:44497 (local address 127.12.244.253:35731)
0926 02:20:30.156443 (+ 18023us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.156453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.167561 (+ 11108us) server_negotiation.cc:407] Connection header received
0926 02:20:30.167646 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.167653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.167807 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.167919 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.171676 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.172895 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.176618 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.177413 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.177567 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.184358 (+ 6791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.184366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.184371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.184432 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.189058 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.190602 (+ 1544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.190609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.190613 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.190786 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.191337 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.191344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.191348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.191786 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.191943 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.194899 (+ 2956us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.195212 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17837,"thread_start_us":17711,"threads_started":1}
I20260926 02:20:30.196830 20584 heartbeater.cc:347] Connected to a master server at 127.12.244.253:35731
I20260926 02:20:30.196928 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.130620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.12.244.195:36613)
0926 02:20:30.167139 (+ 36519us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.167150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.168529 (+ 1379us) server_negotiation.cc:407] Connection header received
0926 02:20:30.168626 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.168639 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.168953 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.169141 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.170930 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.172807 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.175047 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.176126 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.176312 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.187707 (+ 11395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.187731 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.187755 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.187810 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.192358 (+ 4548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.193410 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.193421 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.193424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.193609 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.194136 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.194140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.194144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.194584 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.194893 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.196257 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.196536 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36265,"thread_start_us":36504,"threads_started":1}
I20260926 02:20:30.197206 20584 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:30.198195 20584 heartbeater.cc:511] Master 127.12.244.253:35731 requested a full tablet report, sending...
I20260926 02:20:30.198493 19908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:30.200206 20063 ts_manager.cc:194] Registered new tserver with Master: f6a302db476745678497bc364527197a (127.12.244.194:39225)
I20260926 02:20:30.200627 20583 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33863
I20260926 02:20:30.200796 20583 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:30.200909 20582 heartbeater.cc:347] Connected to a master server at 127.12.244.252:32769
I20260926 02:20:30.201129 20582 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:30.201510 20583 heartbeater.cc:511] Master 127.12.244.254:33863 requested a full tablet report, sending...
I20260926 02:20:30.201779 20582 heartbeater.cc:511] Master 127.12.244.252:32769 requested a full tablet report, sending...
I20260926 02:20:30.202847 19450 ts_manager.cc:194] Re-registered known tserver with Master: f6a302db476745678497bc364527197a (127.12.244.194:39225)
I20260926 02:20:30.204386 19450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:47811
I20260926 02:20:30.205727 20253 ts_manager.cc:194] Registered new tserver with Master: f6a302db476745678497bc364527197a (127.12.244.194:39225)
I20260926 02:20:30.292496 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 19784
W20260926 02:20:30.308439 19435 connection.cc:535] server connection from 127.12.244.195:59121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:20:30.309160 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:36613
--local_ip_for_outbound_sockets=127.12.244.195
--tserver_master_addrs=127.12.244.254:33863
--webserver_port=36215
--webserver_interface=127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab
--principal=oryx/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:20:30.530470 20434 heartbeater.cc:503] Master 127.12.244.254:33863 was elected leader, sending a full tablet report...
W20260926 02:20:30.664448 20595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:30.664745 20595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:30.664845 20595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:30.672356 20595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:30.672505 20595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:30.672621 20595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:30.682674 20595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab
--principal=oryx/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:36613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=36215
--tserver_master_addrs=127.12.244.254:33863,127.12.244.253:35731,127.12.244.252:32769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:30.684154 20595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:30.685542 20595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:30.701776 20600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:30.701839 20603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:30.701700 20601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:30.703629 20595 server_base.cc:1034] running on GCE node
I20260926 02:20:30.704458 20595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:30.705554 20595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:30.706779 20595 hybrid_clock.cc:648] HybridClock initialized: now 1790389230706698 us; error 68 us; skew 500 ppm
I20260926 02:20:30.707165 20595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:30.710733 20595 init.cc:377] Logged in from keytab as oryx/127.12.244.195@KRBTEST.COM (short username oryx)
I20260926 02:20:30.712354 20595 webserver.cc:460] Webserver started at http://127.12.244.195:36215/ using document root <none> and password file <none>
I20260926 02:20:30.713155 20595 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:30.713326 20595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:30.719470 20595 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:30.722637 20610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:30.724351 20595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:20:30.724570 20595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "67b6992471b24ce0b726b833157df909"
format_stamp: "Formatted at 2026-09-26 02:20:27 on dist-test-slave-qfmw"
server_key: "2c24ce633c5725f000650f32eb7bb266"
server_key_iv: "1e6375b3d8116cc47a29037ee80d5bff"
server_key_version: "encryptionkey@0"
I20260926 02:20:30.725694 20595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:30.744704 20595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:30.749475 20595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:30.749835 20595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:30.750959 20595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:30.752416 20595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:30.752540 20595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:30.752681 20595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:30.752769 20595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:30.787541 20595 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:36613
I20260926 02:20:30.787735 20723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:36613 every 8 connection(s)
I20260926 02:20:30.789112 20595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:30.793351 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 20595
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.253@KRBTEST.COM
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:30.857121 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.792049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:46153 (local address 127.12.244.253:35731)
0926 02:20:30.804409 (+ 12360us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.804421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.810810 (+ 6389us) server_negotiation.cc:407] Connection header received
0926 02:20:30.815796 (+ 4986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.815806 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.815932 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.816071 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.819578 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.820856 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.827350 (+ 6494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.828323 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.828472 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.843037 (+ 14565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.843048 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.843055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.843136 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.847936 (+ 4800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.849374 (+ 1438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.849381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.849384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.849556 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.850156 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.850164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.850170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.850623 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.850789 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.856398 (+ 5609us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.856715 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12173,"thread_start_us":12393,"threads_started":1}
Sep 26 02:20:30 dist-test-slave-qfmw krb5kdc[19406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389230, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.252@KRBTEST.COM
I20260926 02:20:30.860675 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.791997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:42117 (local address 127.12.244.254:33863)
0926 02:20:30.804909 (+ 12912us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.804913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.810729 (+ 5816us) server_negotiation.cc:407] Connection header received
0926 02:20:30.815221 (+ 4492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.815231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.815360 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.815507 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.825276 (+ 9769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.826801 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.843354 (+ 16553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.844048 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.844199 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.850683 (+ 6484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.850690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.850694 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.850744 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.854646 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.855884 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.855891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.855894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.856071 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.857686 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.857693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.857697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.858118 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.858293 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.859978 (+ 1685us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.860252 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12768,"thread_start_us":12687,"threads_started":1}
I20260926 02:20:30.862382 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.792747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:42117)
0926 02:20:30.810456 (+ 17709us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.810540 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.814481 (+ 3941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.822270 (+ 7789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.822279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.823037 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:30.823506 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.823524 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.827709 (+ 4185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.827719 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.843194 (+ 15475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.843211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.844563 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.844573 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.844851 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.846550 (+ 1699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:30.846584 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:30.850438 (+ 3854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:30.854996 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.855003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.855010 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.855739 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.856751 (+ 1012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.856760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.856766 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.856971 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.859413 (+ 2442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:30.859421 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:30.859644 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.859832 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.860247 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12583,"mutex_wait_us":6127,"spinlock_wait_cycles":2816,"thread_start_us":13042,"threads_started":1}
I20260926 02:20:30.862915 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.792747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.12.244.195:46153)
0926 02:20:30.810457 (+ 17710us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.810540 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.814659 (+ 4119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.816527 (+ 1868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.816547 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.817905 (+ 1358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:30.819229 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.819272 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.821210 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.821219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.824076 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.824099 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.829019 (+ 4920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.829029 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.831330 (+ 2301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.832762 (+ 1432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:30.832819 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:30.842653 (+ 9834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:30.848237 (+ 5584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.848251 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.848278 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.849200 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.849707 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.849712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.849715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.849992 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.850878 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:30.850894 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:30.855321 (+ 4427us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.855761 (+ 440us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.858930 (+ 3169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13419,"mutex_wait_us":4448,"spinlock_wait_cycles":74368,"thread_start_us":9696,"threads_started":1}
I20260926 02:20:30.864645 20726 heartbeater.cc:347] Connected to a master server at 127.12.244.253:35731
I20260926 02:20:30.865000 20726 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:30.865898 20726 heartbeater.cc:511] Master 127.12.244.253:35731 requested a full tablet report, sending...
I20260926 02:20:30.867071 20725 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33863
I20260926 02:20:30.867262 20725 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:30.867760 20725 heartbeater.cc:511] Master 127.12.244.254:33863 requested a full tablet report, sending...
I20260926 02:20:30.868217 20063 ts_manager.cc:194] Registered new tserver with Master: 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:30.869058 19450 ts_manager.cc:194] Re-registered known tserver with Master: 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:30.870784 19450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:42117
I20260926 02:20:30.873001 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.805904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.12.244.195:40155)
0926 02:20:30.837738 (+ 31834us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.837768 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.838599 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.839554 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.839561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.840255 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:30.843648 (+ 3393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.843659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.850624 (+ 6965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.850630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.852585 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.852608 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.854323 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.854332 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.854607 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.855997 (+ 1390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:30.856028 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:30.863914 (+ 7886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:30.869480 (+ 5566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.869489 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.869496 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.870226 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.870845 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:30.870850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:30.870855 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:30.871058 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:30.871918 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:30.871922 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:30.872079 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.872241 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.872589 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31638,"thread_start_us":31451,"threads_started":1}
I20260926 02:20:30.873951 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.805760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:40155 (local address 127.12.244.252:32769)
0926 02:20:30.812066 (+ 6306us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.812071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.837906 (+ 25835us) server_negotiation.cc:407] Connection header received
0926 02:20:30.838996 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.839006 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.839134 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.839238 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.843841 (+ 4603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.846116 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.852833 (+ 6717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.853785 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.853915 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.864316 (+ 10401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.864327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.864335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.864409 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.868883 (+ 4474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.870460 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.870470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.870477 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.870683 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.871253 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.871261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.871267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.871684 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.871843 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.873357 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.873592 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6177,"thread_start_us":6070,"threads_started":1}
I20260926 02:20:30.874948 20724 heartbeater.cc:347] Connected to a master server at 127.12.244.252:32769
I20260926 02:20:30.875111 20724 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:30.875814 20724 heartbeater.cc:511] Master 127.12.244.252:32769 requested a full tablet report, sending...
I20260926 02:20:30.877146 20253 ts_manager.cc:194] Registered new tserver with Master: 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:30.881214 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.815246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.12.244.253:35731)
0926 02:20:30.861364 (+ 46118us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.861371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.861393 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:20:30.861456 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.861462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.861593 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.861700 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.864661 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.865759 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.867713 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.868341 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.868463 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.872749 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.872757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.872761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.872804 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.877207 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.878168 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.878174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.878178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.878346 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.878841 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.878854 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.878857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.879257 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.879514 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.880150 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.880804 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44508,"thread_start_us":59119,"threads_started":1}
I20260926 02:20:30.885114 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.823036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.12.244.252:32769)
0926 02:20:30.849550 (+ 26514us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.849556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.852817 (+ 3261us) server_negotiation.cc:407] Connection header received
0926 02:20:30.852890 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.852898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.853016 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.853119 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.855879 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.857120 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.872419 (+ 15299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.873385 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.873521 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.878185 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.878192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.878196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.878236 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.882000 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.882826 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.882835 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.882841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.883028 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.883520 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.883524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.883526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.883866 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.884124 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.884501 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.884676 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26384,"thread_start_us":26779,"threads_started":1}
I20260926 02:20:30.888031 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.814321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.12.244.254:33863)
0926 02:20:30.859806 (+ 45485us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.859811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.859828 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:20:30.859884 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.859889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.860116 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.869223 (+ 9107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:30.869422 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.875496 (+ 6074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.877693 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.878309 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.878449 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.882142 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:30.882147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:30.882150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:30.882185 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:30.885464 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.886132 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.886138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.886142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.886338 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:30.886722 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:30.886726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:30.886728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:30.886995 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:30.887371 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.887586 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.887759 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5133,"server-negotiator.queue_time_us":45355,"thread_start_us":45624,"threads_started":1}
I20260926 02:20:30.892675 19450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:20:30.900488 19450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:30.947918 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.921298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45503 (local address 127.0.0.1:34176)
0926 02:20:30.936552 (+ 15254us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.936582 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.936708 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.937707 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.937715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.937751 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:30.938321 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.938336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.941455 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.941464 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.944203 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.944223 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.945884 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.945891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.946204 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.946233 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.946356 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.946509 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15123,"mutex_wait_us":27,"thread_start_us":15603,"threads_started":1}
I20260926 02:20:30.947381 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.921507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.12.244.193:45503)
0926 02:20:30.935915 (+ 14408us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.935926 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.936705 (+ 779us) server_negotiation.cc:407] Connection header received
0926 02:20:30.936932 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.936942 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.937486 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.938588 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:30.938836 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.941254 (+ 2418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.944406 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.945655 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.945867 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.946086 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.946423 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.946591 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14199,"thread_start_us":14428,"threads_started":1}
I20260926 02:20:30.948856 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.921059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.12.244.194:39225)
0926 02:20:30.935914 (+ 14855us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.935926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.935964 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:20:30.936083 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.936091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.936433 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.936701 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:30.937330 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.939352 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.945481 (+ 6129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.947662 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.947873 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.948182 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.948278 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.948443 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14670,"thread_start_us":14580,"threads_started":1}
I20260926 02:20:30.949397 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.920784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:39225 (local address 127.0.0.1:53870)
0926 02:20:30.921583 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.921609 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.921771 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.936725 (+ 14954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.936742 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.936774 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:30.937125 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.937137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.943444 (+ 6307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.943450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.945268 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.945282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.947923 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.947929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.948105 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.948138 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.948224 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.948383 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":1,"thread_start_us":649,"threads_started":1}
I20260926 02:20:30.953192 20367 tablet_service.cc:1467] Processing CreateTablet for tablet 74d80bb06d7c4ea395d468a0d33d0b66 (DEFAULT_TABLE table=test-table [id=eed9007e96eb48d5814cd60a395ac9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:30.953445 20515 tablet_service.cc:1467] Processing CreateTablet for tablet 74d80bb06d7c4ea395d468a0d33d0b66 (DEFAULT_TABLE table=test-table [id=eed9007e96eb48d5814cd60a395ac9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:30.954596 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d80bb06d7c4ea395d468a0d33d0b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:30.954782 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d80bb06d7c4ea395d468a0d33d0b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:30.960842 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.937282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:36613 (local address 127.0.0.1:50436)
0926 02:20:30.939914 (+ 2632us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:30.939947 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:30.940035 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:30.952418 (+ 12383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:30.952424 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:30.952461 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:30.952796 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.952809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.954991 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.954997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.957150 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:30.957171 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.959808 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.959818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:30.960029 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.960070 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:20:30.960278 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 02:20:30.960456 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2508,"thread_start_us":2368,"threads_started":1}
I20260926 02:20:30.960878 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:30.937552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.12.244.195:36613)
0926 02:20:30.951729 (+ 14177us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:30.951739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:30.951774 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:20:30.951876 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:30.951884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:30.952146 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:30.952378 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:30.953009 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.954767 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.957375 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:30.959017 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:30.959258 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:30.959532 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:30.960271 (+ 739us) server_negotiation.cc:299] Negotiation successful
0926 02:20:30.960529 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13988,"thread_start_us":13803,"threads_started":1}
I20260926 02:20:30.965730 20658 tablet_service.cc:1467] Processing CreateTablet for tablet 74d80bb06d7c4ea395d468a0d33d0b66 (DEFAULT_TABLE table=test-table [id=eed9007e96eb48d5814cd60a395ac9db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:30.967341 20658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d80bb06d7c4ea395d468a0d33d0b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:30.969961 20762 tablet_bootstrap.cc:492] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Bootstrap starting.
I20260926 02:20:30.972256 20763 tablet_bootstrap.cc:492] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Bootstrap starting.
I20260926 02:20:30.974920 20762 tablet_bootstrap.cc:654] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:30.975854 20763 tablet_bootstrap.cc:654] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:30.976377 20762 log.cc:824] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:30.977174 20763 log.cc:824] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:30.980316 20762 tablet_bootstrap.cc:492] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: No bootstrap required, opened a new log
I20260926 02:20:30.980679 20762 ts_tablet_manager.cc:1397] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260926 02:20:30.982487 20766 tablet_bootstrap.cc:492] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Bootstrap starting.
I20260926 02:20:30.983326 20763 tablet_bootstrap.cc:492] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: No bootstrap required, opened a new log
I20260926 02:20:30.983701 20763 ts_tablet_manager.cc:1397] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260926 02:20:30.986433 20766 tablet_bootstrap.cc:654] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:30.987914 20766 log.cc:824] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:30.990863 20766 tablet_bootstrap.cc:492] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: No bootstrap required, opened a new log
I20260926 02:20:30.991240 20766 ts_tablet_manager.cc:1397] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:20:30.992698 20762 raft_consensus.cc:348] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:30.993235 20762 raft_consensus.cc:374] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:30.993359 20762 raft_consensus.cc:729] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f283717ddaad49cd9d5fdfde6ef2d2fe, State: Initialized, Role: FOLLOWER
I20260926 02:20:30.994144 20762 consensus_queue.cc:260] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:30.995652 20763 raft_consensus.cc:348] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:30.996115 20762 ts_tablet_manager.cc:1428] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260926 02:20:30.996127 20763 raft_consensus.cc:374] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:30.996219 20763 raft_consensus.cc:729] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a302db476745678497bc364527197a, State: Initialized, Role: FOLLOWER
I20260926 02:20:30.996861 20763 consensus_queue.cc:260] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:31.002012 20766 raft_consensus.cc:348] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:31.002530 20766 raft_consensus.cc:374] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:31.002650 20766 raft_consensus.cc:729] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b6992471b24ce0b726b833157df909, State: Initialized, Role: FOLLOWER
I20260926 02:20:31.004225 20763 ts_tablet_manager.cc:1428] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260926 02:20:31.004727 20583 heartbeater.cc:503] Master 127.12.244.254:33863 was elected leader, sending a full tablet report...
I20260926 02:20:31.003438 20766 consensus_queue.cc:260] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:31.009845 20766 ts_tablet_manager.cc:1428] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
I20260926 02:20:31.010236 20725 heartbeater.cc:503] Master 127.12.244.254:33863 was elected leader, sending a full tablet report...
W20260926 02:20:31.042217 20727 tablet.cc:2369] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:20:31.170156 20587 tablet.cc:2369] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:20:31.206583 20768 raft_consensus.cc:482] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:31.206990 20768 raft_consensus.cc:504] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:31.208945 20768 leader_election.cc:288] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6a302db476745678497bc364527197a (127.12.244.194:39225), 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:31.216187 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.209369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:41231 (local address 127.12.244.194:39225)
0926 02:20:31.209611 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:31.209621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:31.210459 (+ 838us) server_negotiation.cc:407] Connection header received
0926 02:20:31.210648 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:31.210655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:31.210781 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:31.210884 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:31.211513 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.212186 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.214061 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.214869 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.215048 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.215153 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:31.215761 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 02:20:31.215905 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:20:31.216213 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.209403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:35419 (local address 127.12.244.195:36613)
0926 02:20:31.209659 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:31.209668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:31.210355 (+ 687us) server_negotiation.cc:407] Connection header received
0926 02:20:31.210582 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:31.210586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:31.210646 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:31.210749 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:31.211447 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.212190 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.214080 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.214850 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.215048 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.215151 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:31.215770 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 02:20:31.215971 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:20:31.216393 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.209211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:36613 (local address 127.12.244.193:35419)
0926 02:20:31.210223 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:31.210242 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:31.210430 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:31.210768 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:31.210773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:31.210806 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:31.211293 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:31.211304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.212332 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.212339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:31.213905 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:31.213918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.215086 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.215121 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:31.215511 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.215627 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 02:20:31.215741 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:20:31.215911 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":111,"thread_start_us":685,"threads_started":1}
I20260926 02:20:31.216413 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.209213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:39225 (local address 127.12.244.193:41231)
0926 02:20:31.210343 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:31.210358 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:31.210505 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:31.210898 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:31.210906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:31.210952 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:31.211359 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:31.211368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.212329 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.212336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:31.213908 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:31.213920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.215352 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.215359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:31.215529 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.215632 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 02:20:31.215735 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:20:31.215893 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1022,"mutex_wait_us":196,"thread_start_us":551,"threads_started":1}
I20260926 02:20:31.218010 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d80bb06d7c4ea395d468a0d33d0b66" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a302db476745678497bc364527197a" is_pre_election: true
I20260926 02:20:31.217533 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d80bb06d7c4ea395d468a0d33d0b66" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b6992471b24ce0b726b833157df909" is_pre_election: true
I20260926 02:20:31.218520 20536 raft_consensus.cc:2454] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 0.
I20260926 02:20:31.218532 20679 raft_consensus.cc:2454] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 0.
I20260926 02:20:31.219311 20323 leader_election.cc:302] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67b6992471b24ce0b726b833157df909, f283717ddaad49cd9d5fdfde6ef2d2fe, f6a302db476745678497bc364527197a; no voters:
I20260926 02:20:31.219758 20768 raft_consensus.cc:2781] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:31.219928 20768 raft_consensus.cc:482] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:31.220041 20768 raft_consensus.cc:3037] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:31.222504 20768 raft_consensus.cc:504] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:31.223397 20768 leader_election.cc:288] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 election: Requested vote from peers f6a302db476745678497bc364527197a (127.12.244.194:39225), 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:31.224054 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d80bb06d7c4ea395d468a0d33d0b66" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a302db476745678497bc364527197a"
I20260926 02:20:31.224067 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d80bb06d7c4ea395d468a0d33d0b66" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b6992471b24ce0b726b833157df909"
I20260926 02:20:31.224305 20536 raft_consensus.cc:3037] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:31.224334 20679 raft_consensus.cc:3037] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:31.227679 20679 raft_consensus.cc:2454] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 1.
I20260926 02:20:31.227679 20536 raft_consensus.cc:2454] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 1.
I20260926 02:20:31.228273 20322 leader_election.cc:302] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67b6992471b24ce0b726b833157df909, f283717ddaad49cd9d5fdfde6ef2d2fe, f6a302db476745678497bc364527197a; no voters:
W20260926 02:20:31.228547 20436 tablet.cc:2369] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:20:31.228668 20768 raft_consensus.cc:2781] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:31.229245 20768 raft_consensus.cc:686] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 LEADER]: Becoming Leader. State: Replica: f283717ddaad49cd9d5fdfde6ef2d2fe, State: Running, Role: LEADER
I20260926 02:20:31.230041 20768 consensus_queue.cc:237] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:31.235919 19451 catalog_manager.cc:5495] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193). New cstate: current_term: 1 leader_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } health_report { overall_health: UNKNOWN } } }
I20260926 02:20:31.316231 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.311024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.12.244.193:45503)
0926 02:20:31.311250 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:31.311254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:31.311271 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:20:31.311361 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:31.311365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:31.311451 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:31.311581 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:31.311923 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.312668 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.314136 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.314573 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.314681 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.315183 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:31.315744 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:31.315862 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:31.315959 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 02:20:31.316026 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:20:31.325105 20536 raft_consensus.cc:1261] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Refusing update from remote peer f283717ddaad49cd9d5fdfde6ef2d2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:31.325167 20679 raft_consensus.cc:1261] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Refusing update from remote peer f283717ddaad49cd9d5fdfde6ef2d2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:31.326221 20773 consensus_queue.cc:1035] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:31.326639 20768 consensus_queue.cc:1035] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:31.349786 20776 mvcc.cc:205] Tried to move back new op lower bound from 7333434291496288256 to 7333434291122720768. Current Snapshot: MvccSnapshot[applied={T|T < 7333434291496288256}]
I20260926 02:20:31.367908 19451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:31.368216 19451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:31.374951 19451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4926ed7f7fde4132b6b7a411a01bb9df: Sending DeleteTablet for 3 replicas of tablet 74d80bb06d7c4ea395d468a0d33d0b66
I20260926 02:20:31.376493 20515 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d80bb06d7c4ea395d468a0d33d0b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:31 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:53870
I20260926 02:20:31.376596 20367 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d80bb06d7c4ea395d468a0d33d0b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:31 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:34176
I20260926 02:20:31.377640 20787 tablet_replica.cc:330] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: stopping tablet replica
I20260926 02:20:31.378121 20787 raft_consensus.cc:2229] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:31.378093 20658 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d80bb06d7c4ea395d468a0d33d0b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:31 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:50436
I20260926 02:20:31.378909 20787 raft_consensus.cc:2258] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:31.379869 20786 tablet_replica.cc:330] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: stopping tablet replica
I20260926 02:20:31.380720 20786 raft_consensus.cc:2229] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:31.381330 20786 raft_consensus.cc:2258] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:31.383500 20787 ts_tablet_manager.cc:1905] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:31.386170 20786 ts_tablet_manager.cc:1905] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:31.392899 20787 ts_tablet_manager.cc:1918] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:31.393124 20787 log.cc:1196] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/74d80bb06d7c4ea395d468a0d33d0b66
I20260926 02:20:31.393761 20787 ts_tablet_manager.cc:1939] T 74d80bb06d7c4ea395d468a0d33d0b66 P f283717ddaad49cd9d5fdfde6ef2d2fe: Deleting consensus metadata
I20260926 02:20:31.395407 20788 tablet_replica.cc:330] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: stopping tablet replica
I20260926 02:20:31.395897 20788 raft_consensus.cc:2229] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:31.396399 20788 raft_consensus.cc:2258] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:31.397471 20786 ts_tablet_manager.cc:1918] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:31.397693 20786 log.cc:1196] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/74d80bb06d7c4ea395d468a0d33d0b66
I20260926 02:20:31.398424 20786 ts_tablet_manager.cc:1939] T 74d80bb06d7c4ea395d468a0d33d0b66 P f6a302db476745678497bc364527197a: Deleting consensus metadata
I20260926 02:20:31.398836 20788 ts_tablet_manager.cc:1905] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:31.400804 19438 catalog_manager.cc:4841] TS f6a302db476745678497bc364527197a (127.12.244.194:39225): tablet 74d80bb06d7c4ea395d468a0d33d0b66 (table test-table [id=eed9007e96eb48d5814cd60a395ac9db]) successfully deleted
I20260926 02:20:31.404721 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 19419
I20260926 02:20:31.405810 19437 catalog_manager.cc:4841] TS f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193:45503): tablet 74d80bb06d7c4ea395d468a0d33d0b66 (table test-table [id=eed9007e96eb48d5814cd60a395ac9db]) successfully deleted
I20260926 02:20:31.406337 19482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:31.411972 20788 ts_tablet_manager.cc:1918] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:31.412191 20788 log.cc:1196] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/74d80bb06d7c4ea395d468a0d33d0b66
I20260926 02:20:31.412791 20788 ts_tablet_manager.cc:1939] T 74d80bb06d7c4ea395d468a0d33d0b66 P 67b6992471b24ce0b726b833157df909: Deleting consensus metadata
I20260926 02:20:31.430202 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.386160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.12.244.253:35731)
0926 02:20:31.402423 (+ 16263us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:31.402434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:31.408546 (+ 6112us) server_negotiation.cc:407] Connection header received
0926 02:20:31.408657 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:31.408665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:31.409191 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:31.409381 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:31.412947 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.414307 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.417013 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.417772 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.418029 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.422304 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:31.422313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:31.422319 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:31.422374 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:31.426669 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.427676 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.427684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.427690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.427899 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.428343 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.428347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.428350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.428740 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:31.429096 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:31.429416 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 02:20:31.429720 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16080,"thread_start_us":15973,"threads_started":1}
I20260926 02:20:31.449034 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.403624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.12.244.252:32769)
0926 02:20:31.417368 (+ 13744us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:31.417375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:31.430256 (+ 12881us) server_negotiation.cc:407] Connection header received
0926 02:20:31.430534 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:31.430541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:31.430647 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:31.430723 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:31.433874 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.435118 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.436985 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.437651 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.437795 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.441422 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:31.441429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:31.441435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:31.441479 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:31.445308 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.446617 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.446623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.446627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.446792 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.447259 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.447264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.447267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.447684 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:31.447982 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:31.448344 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 02:20:31.448585 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13597,"thread_start_us":14537,"threads_started":1}
I20260926 02:20:31.554993 19437 catalog_manager.cc:4841] TS 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613): tablet 74d80bb06d7c4ea395d468a0d33d0b66 (table test-table [id=eed9007e96eb48d5814cd60a395ac9db]) successfully deleted
I20260926 02:20:31.557715 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 19419
I20260926 02:20:31.611307 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.585806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.12.244.252:32769)
0926 02:20:31.585976 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:31.585982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:31.590502 (+ 4520us) server_negotiation.cc:407] Connection header received
0926 02:20:31.591191 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:31.591200 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:31.591352 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:31.591457 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:31.595612 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.596850 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.598997 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.599691 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.599794 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.604230 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:31.604239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:31.604246 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:31.604298 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:31.608037 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.609004 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.609010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.609014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.609166 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.609684 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.609693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.609698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.610101 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:31.610350 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:31.610708 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 02:20:31.610943 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:20:31.612746 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:31.585221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.12.244.253:35731)
0926 02:20:31.585461 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:31.585471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:31.595032 (+ 9561us) server_negotiation.cc:407] Connection header received
0926 02:20:31.595346 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:31.595353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:31.595439 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:31.595515 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:31.598036 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.599078 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.601155 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:31.601878 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:31.601987 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:31.605645 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:31.605652 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:31.605658 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:31.605703 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:31.609145 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.610033 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.610038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.610042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.610196 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:31.610751 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:31.610756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:31.610760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:31.611156 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:31.611390 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:31.612145 (+ 755us) server_negotiation.cc:299] Negotiation successful
0926 02:20:31.612377 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:20:32.347196 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.346210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:54311)
0926 02:20:32.346747 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.346950 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":313,"threads_started":1}
W20260926 02:20:32.347671 20725 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:33863 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.358863 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.357856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:47027)
0926 02:20:32.358455 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.358619 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":340,"threads_started":1}
W20260926 02:20:32.359833 20583 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:33863 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.362819 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.361949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:39677)
0926 02:20:32.362525 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.362637 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":311,"threads_started":1}
W20260926 02:20:32.363521 20434 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:33863 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.876356 20803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4926ed7f7fde4132b6b7a411a01bb9df)
I20260926 02:20:32.876706 20804 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4926ed7f7fde4132b6b7a411a01bb9df)
I20260926 02:20:32.876657 20803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:32.876940 20804 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:32.877614 20803 leader_election.cc:288] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863), c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253:35731)
I20260926 02:20:32.878943 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.877756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53002)
0926 02:20:32.878357 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.878583 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":272,"threads_started":1}
W20260926 02:20:32.879432 20239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.887715 20804 leader_election.cc:288] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863), 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252:32769)
I20260926 02:20:32.892905 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.878190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53018)
0926 02:20:32.892180 (+ 13990us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.892496 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13822,"thread_start_us":13665,"threads_started":1}
W20260926 02:20:32.893566 20050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.897866 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.891765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53032)
0926 02:20:32.897319 (+ 5554us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.897561 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5376}
W20260926 02:20:32.899293 20239 leader_election.cc:334] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863): Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.900534 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.877850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35731 (local address 127.0.0.1:39324)
0926 02:20:32.885458 (+ 7608us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.885484 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:32.885641 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:32.893839 (+ 8198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:32.893844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:32.893873 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:32.894079 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:32.894089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.895966 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.895970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:32.898036 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:32.898051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.899688 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.899695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:32.899874 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:32.899914 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:20:32.899996 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:20:32.900161 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7458,"thread_start_us":21110,"threads_started":1}
I20260926 02:20:32.901328 20807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.877910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:32769 (local address 127.0.0.1:49030)
0926 02:20:32.884954 (+ 7044us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.884987 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:32.885206 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:32.892877 (+ 7671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:32.892882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:32.892919 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:32.893289 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:32.893308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.895082 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.895109 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:32.897288 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:32.897304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.900354 (+ 3050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.900360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:32.900602 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:32.900644 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:20:32.900736 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:20:32.900911 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6832,"thread_start_us":6617,"threads_started":1}
I20260926 02:20:32.901489 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.878326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.12.244.252:32769)
0926 02:20:32.892467 (+ 14141us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:32.892477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:32.892503 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:20:32.892597 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:32.892606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:32.892711 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:32.892843 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:32.893537 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.894920 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.897493 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.900010 (+ 2517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.900145 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:32.900211 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:32.900970 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 02:20:32.901127 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13933,"thread_start_us":14810,"threads_started":1}
I20260926 02:20:32.901402 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.878559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.12.244.253:35731)
0926 02:20:32.893500 (+ 14941us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:32.893508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:32.893522 (+ 14us) server_negotiation.cc:407] Connection header received
0926 02:20:32.893577 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:32.893581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:32.893661 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:32.894229 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:32.894502 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.895781 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.898237 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:32.899507 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:32.900822 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:32.900891 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:32.900939 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 02:20:32.901079 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14760,"thread_start_us":19471,"threads_started":1}
I20260926 02:20:32.901911 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.901548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53056)
0926 02:20:32.901726 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.901795 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:20:32.902101 20050 leader_election.cc:334] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863): Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.902393 20073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14b6519af9cc47d7a99b11fab42854b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" is_pre_election: true
I20260926 02:20:32.902541 20263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "14b6519af9cc47d7a99b11fab42854b1" is_pre_election: true
I20260926 02:20:32.902680 20073 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6519af9cc47d7a99b11fab42854b1 in term 1.
I20260926 02:20:32.902918 20263 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b2df03947a42809d10cbfcbbeb1bdb in term 1.
I20260926 02:20:32.903170 20236 leader_election.cc:302] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14b6519af9cc47d7a99b11fab42854b1, c8b2df03947a42809d10cbfcbbeb1bdb; no voters: 4926ed7f7fde4132b6b7a411a01bb9df
I20260926 02:20:32.903561 20803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 02:20:32.903481 20049 leader_election.cc:302] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14b6519af9cc47d7a99b11fab42854b1, c8b2df03947a42809d10cbfcbbeb1bdb; no voters: 4926ed7f7fde4132b6b7a411a01bb9df
I20260926 02:20:32.903680 20803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4926ed7f7fde4132b6b7a411a01bb9df)
I20260926 02:20:32.903774 20803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:20:32.903821 20804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 02:20:32.903946 20804 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4926ed7f7fde4132b6b7a411a01bb9df)
I20260926 02:20:32.904065 20804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:20:32.906478 20804 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:32.906498 20803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:32.907183 20804 leader_election.cc:288] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 2 election: Requested vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863), 14b6519af9cc47d7a99b11fab42854b1 (127.12.244.252:32769)
I20260926 02:20:32.907198 20803 leader_election.cc:288] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 election: Requested vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863), c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253:35731)
I20260926 02:20:32.907727 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.907305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53058)
0926 02:20:32.907467 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.907552 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 02:20:32.907837 20263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "14b6519af9cc47d7a99b11fab42854b1"
I20260926 02:20:32.907944 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.907329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53062)
0926 02:20:32.907631 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.907728 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 02:20:32.908144 20263 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c8b2df03947a42809d10cbfcbbeb1bdb in current term 2: Already voted for candidate 14b6519af9cc47d7a99b11fab42854b1 in this term.
I20260926 02:20:32.908308 20073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14b6519af9cc47d7a99b11fab42854b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
I20260926 02:20:32.908586 20073 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 14b6519af9cc47d7a99b11fab42854b1 in current term 2: Already voted for candidate c8b2df03947a42809d10cbfcbbeb1bdb in this term.
I20260926 02:20:32.909222 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.908621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53064)
0926 02:20:32.908998 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.909090 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
I20260926 02:20:32.909366 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:32.909008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53078)
0926 02:20:32.909143 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:32.909217 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:20:32.909394 20239 leader_election.cc:334] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863): Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.909584 20239 leader_election.cc:302] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14b6519af9cc47d7a99b11fab42854b1; no voters: 4926ed7f7fde4132b6b7a411a01bb9df, c8b2df03947a42809d10cbfcbbeb1bdb
I20260926 02:20:32.909857 20803 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260926 02:20:32.910028 20050 leader_election.cc:334] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863): Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:32.910161 20050 leader_election.cc:302] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8b2df03947a42809d10cbfcbbeb1bdb; no voters: 14b6519af9cc47d7a99b11fab42854b1, 4926ed7f7fde4132b6b7a411a01bb9df
I20260926 02:20:32.910459 20804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 02:20:33.349388 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:33.348526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:46093)
0926 02:20:33.349102 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:33.349219 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":343,"threads_started":1}
I20260926 02:20:33.361546 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:33.360649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:59753)
0926 02:20:33.361213 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:33.361318 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":338,"threads_started":1}
I20260926 02:20:33.364900 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:33.364237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:44335)
0926 02:20:33.364630 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:33.364708 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":209,"threads_started":1}
I20260926 02:20:34.351244 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.350320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:58953)
0926 02:20:34.350882 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.350975 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":293,"threads_started":1}
I20260926 02:20:34.363561 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.362677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:41643)
0926 02:20:34.363232 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.363353 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
I20260926 02:20:34.366576 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.365726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:35821)
0926 02:20:34.366303 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.366399 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":294,"threads_started":1}
I20260926 02:20:34.441665 20819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:34.441910 20819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:34.442646 20819 leader_election.cc:288] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863), c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253:35731)
I20260926 02:20:34.443943 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.442966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53268)
0926 02:20:34.443519 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.443672 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":256,"threads_started":1}
I20260926 02:20:34.443902 20073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14b6519af9cc47d7a99b11fab42854b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" is_pre_election: true
I20260926 02:20:34.444183 20073 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6519af9cc47d7a99b11fab42854b1 in term 2.
I20260926 02:20:34.444679 20236 leader_election.cc:302] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6519af9cc47d7a99b11fab42854b1, c8b2df03947a42809d10cbfcbbeb1bdb; no voters:
I20260926 02:20:34.445000 20819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 02:20:34.445127 20819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:34.445256 20819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:20:34.445422 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.444974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53276)
0926 02:20:34.445138 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.445252 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 02:20:34.445636 20239 leader_election.cc:334] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863): Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:34.447618 20819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:34.448254 20819 leader_election.cc:288] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 3 election: Requested vote from peers 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863), c8b2df03947a42809d10cbfcbbeb1bdb (127.12.244.253:35731)
I20260926 02:20:34.448863 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.448370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53282)
0926 02:20:34.448586 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.448668 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 02:20:34.448971 20073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14b6519af9cc47d7a99b11fab42854b1" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb"
I20260926 02:20:34.449203 20073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:20:34.449978 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.449538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53286)
0926 02:20:34.449751 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.449833 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 02:20:34.450125 20239 leader_election.cc:334] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863): Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
I20260926 02:20:34.451665 20073 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6519af9cc47d7a99b11fab42854b1 in term 3.
I20260926 02:20:34.452250 20236 leader_election.cc:302] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14b6519af9cc47d7a99b11fab42854b1, c8b2df03947a42809d10cbfcbbeb1bdb; no voters: 4926ed7f7fde4132b6b7a411a01bb9df
I20260926 02:20:34.452706 20819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:20:34.453222 20819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [term 3 LEADER]: Becoming Leader. State: Replica: 14b6519af9cc47d7a99b11fab42854b1, State: Running, Role: LEADER
I20260926 02:20:34.453635 20819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } }
I20260926 02:20:34.456084 20822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14b6519af9cc47d7a99b11fab42854b1. Latest consensus state: current_term: 3 leader_uuid: "14b6519af9cc47d7a99b11fab42854b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:34.456386 20822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:34.457121 20824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:34.460827 20824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:34.461623 20824 catalog_manager.cc:1259] Loaded cluster ID: f7663f0c98dd4b1c96e53242122e188c
I20260926 02:20:34.461705 20824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:34.462787 20824 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:34.463693 20824 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Loaded TSK: 0
I20260926 02:20:34.464637 20824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:20:34.509814 20252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:20:34.517274 20252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:34.521906 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.521361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53292)
0926 02:20:34.521595 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.521704 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 02:20:34.521855 20073 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [term 3 FOLLOWER]: Refusing update from remote peer 14b6519af9cc47d7a99b11fab42854b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 02:20:34.522639 20822 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 02:20:34.523046 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.522578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53306)
0926 02:20:34.522798 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.522892 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:20:34.523303 20239 consensus_peers.cc:487] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 -> Peer 4926ed7f7fde4132b6b7a411a01bb9df (127.12.244.254:33863): Couldn't send request to peer 4926ed7f7fde4132b6b7a411a01bb9df. Status: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 02:20:34.525643 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14b6519af9cc47d7a99b11fab42854b1. Latest consensus state: current_term: 3 leader_uuid: "14b6519af9cc47d7a99b11fab42854b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:34.525946 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:20:34.527664 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "14b6519af9cc47d7a99b11fab42854b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:34.528364 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:34.528641 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "14b6519af9cc47d7a99b11fab42854b1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4926ed7f7fde4132b6b7a411a01bb9df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33863 } } peers { permanent_uuid: "c8b2df03947a42809d10cbfcbbeb1bdb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35731 } attrs { promote: false } } peers { permanent_uuid: "14b6519af9cc47d7a99b11fab42854b1" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 32769 } attrs { promote: false } } }
I20260926 02:20:34.528918 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b2df03947a42809d10cbfcbbeb1bdb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:20:34.560168 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.538757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.12.244.194:39225)
0926 02:20:34.549299 (+ 10542us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:34.549312 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:34.551485 (+ 2173us) server_negotiation.cc:407] Connection header received
0926 02:20:34.551592 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:34.551600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:34.551705 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:34.551818 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:34.552978 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.554444 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.557401 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.558857 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.559080 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:34.559206 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:34.559564 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 02:20:34.559733 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9678,"thread_start_us":9479,"threads_started":1}
I20260926 02:20:34.560638 20820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.537522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45503 (local address 127.0.0.1:37104)
0926 02:20:34.537820 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.537849 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:34.537967 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:34.554296 (+ 16329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:34.554302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:34.554335 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:34.554704 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:34.554720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.556443 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.556449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:34.558209 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:34.558220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.559867 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.559874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:34.560041 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:34.560073 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:20:34.560132 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 02:20:34.560289 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":27}
I20260926 02:20:34.561504 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.538043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:39225 (local address 127.0.0.1:42162)
0926 02:20:34.551040 (+ 12997us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.551070 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:34.551318 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:34.552105 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:34.552114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:34.552152 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:34.552708 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:34.552723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.554648 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.554657 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:34.557218 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:34.557229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.559079 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.559119 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:34.559355 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:34.559384 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:20:34.559524 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 02:20:34.561152 (+ 1628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12758,"mutex_wait_us":23,"thread_start_us":12528,"threads_started":1}
I20260926 02:20:34.561546 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.537807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.12.244.193:45503)
0926 02:20:34.553948 (+ 16141us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:34.553953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:34.553974 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:20:34.554041 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:34.554045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:34.554110 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:34.554276 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:34.554913 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.556246 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.558376 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.559658 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.560935 (+ 1277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:34.561007 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:34.561061 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 02:20:34.561206 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16020,"thread_start_us":16627,"threads_started":1}
I20260926 02:20:34.562553 20515 tablet_service.cc:1467] Processing CreateTablet for tablet 7c141e815df44afaa1f85af704977c08 (DEFAULT_TABLE table=test-table [id=c7974f41ebe348e080e46bbb99f5a74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:34.562548 20367 tablet_service.cc:1467] Processing CreateTablet for tablet 7c141e815df44afaa1f85af704977c08 (DEFAULT_TABLE table=test-table [id=c7974f41ebe348e080e46bbb99f5a74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:34.563045 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c141e815df44afaa1f85af704977c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:34.563048 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c141e815df44afaa1f85af704977c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:34.563680 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.538558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.12.244.195:36613)
0926 02:20:34.550575 (+ 12017us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:34.550583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:34.555649 (+ 5066us) server_negotiation.cc:407] Connection header received
0926 02:20:34.556044 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:34.556050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:34.556126 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:34.556194 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:34.557181 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.558489 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.561132 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.562363 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.562478 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:34.562545 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:34.563205 (+ 660us) server_negotiation.cc:299] Negotiation successful
0926 02:20:34.563366 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11818,"thread_start_us":11636,"threads_started":1}
I20260926 02:20:34.564378 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.538382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:36613 (local address 127.0.0.1:54292)
0926 02:20:34.555519 (+ 17137us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.555539 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:34.555682 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:34.556408 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:34.556412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:34.556434 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:34.556962 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:34.556974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.558638 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.558643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:34.560962 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:34.560975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:34.562744 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:34.562750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:34.562981 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:34.563015 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:20:34.563928 (+ 913us) client_negotiation.cc:240] Negotiation successful
0926 02:20:34.564073 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16962,"thread_start_us":17394,"threads_started":1}
I20260926 02:20:34.565198 20658 tablet_service.cc:1467] Processing CreateTablet for tablet 7c141e815df44afaa1f85af704977c08 (DEFAULT_TABLE table=test-table [id=c7974f41ebe348e080e46bbb99f5a74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:34.565649 20658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c141e815df44afaa1f85af704977c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:34.569733 20837 tablet_bootstrap.cc:492] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: Bootstrap starting.
I20260926 02:20:34.571908 20838 tablet_bootstrap.cc:492] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: Bootstrap starting.
I20260926 02:20:34.573189 20839 tablet_bootstrap.cc:492] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: Bootstrap starting.
I20260926 02:20:34.573999 20837 tablet_bootstrap.cc:654] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:34.574060 20838 tablet_bootstrap.cc:654] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:34.574671 20839 tablet_bootstrap.cc:654] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:34.576815 20838 tablet_bootstrap.cc:492] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: No bootstrap required, opened a new log
I20260926 02:20:34.577039 20838 ts_tablet_manager.cc:1397] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 02:20:34.577294 20837 tablet_bootstrap.cc:492] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: No bootstrap required, opened a new log
I20260926 02:20:34.577459 20837 ts_tablet_manager.cc:1397] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260926 02:20:34.577759 20839 tablet_bootstrap.cc:492] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: No bootstrap required, opened a new log
I20260926 02:20:34.577899 20839 ts_tablet_manager.cc:1397] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 02:20:34.578007 20838 raft_consensus.cc:348] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.578267 20838 raft_consensus.cc:374] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:34.578379 20838 raft_consensus.cc:729] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f283717ddaad49cd9d5fdfde6ef2d2fe, State: Initialized, Role: FOLLOWER
I20260926 02:20:34.578300 20837 raft_consensus.cc:348] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.578534 20837 raft_consensus.cc:374] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:34.578639 20837 raft_consensus.cc:729] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a302db476745678497bc364527197a, State: Initialized, Role: FOLLOWER
I20260926 02:20:34.578681 20839 raft_consensus.cc:348] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.578907 20839 raft_consensus.cc:374] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:34.578748 20838 consensus_queue.cc:260] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.579000 20839 raft_consensus.cc:729] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b6992471b24ce0b726b833157df909, State: Initialized, Role: FOLLOWER
I20260926 02:20:34.578908 20837 consensus_queue.cc:260] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.579305 20839 consensus_queue.cc:260] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.583033 20837 ts_tablet_manager.cc:1428] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 02:20:34.583045 20838 ts_tablet_manager.cc:1428] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260926 02:20:34.584167 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.583713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:34485)
0926 02:20:34.583909 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.584006 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:20:34.584304 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.583666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:36067)
0926 02:20:34.583992 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.584101 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260926 02:20:34.585924 20839 ts_tablet_manager.cc:1428] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260926 02:20:34.588119 20582 heartbeater.cc:503] Master 127.12.244.252:32769 was elected leader, sending a full tablet report...
I20260926 02:20:34.588425 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.587058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:36441)
0926 02:20:34.588134 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.588250 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":935}
I20260926 02:20:34.589002 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.588429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:36751)
0926 02:20:34.588675 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.588762 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260926 02:20:34.590587 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.589097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:46991)
0926 02:20:34.590297 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.590401 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1066}
I20260926 02:20:34.590655 20724 heartbeater.cc:503] Master 127.12.244.252:32769 was elected leader, sending a full tablet report...
I20260926 02:20:34.591557 20433 heartbeater.cc:503] Master 127.12.244.252:32769 was elected leader, sending a full tablet report...
I20260926 02:20:34.930939 20843 raft_consensus.cc:482] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:34.931320 20843 raft_consensus.cc:504] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.932057 20843 leader_election.cc:288] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6a302db476745678497bc364527197a (127.12.244.194:39225), 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:34.932636 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c141e815df44afaa1f85af704977c08" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b6992471b24ce0b726b833157df909" is_pre_election: true
I20260926 02:20:34.932644 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c141e815df44afaa1f85af704977c08" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a302db476745678497bc364527197a" is_pre_election: true
I20260926 02:20:34.932935 20536 raft_consensus.cc:2454] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 0.
I20260926 02:20:34.932935 20679 raft_consensus.cc:2454] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 0.
I20260926 02:20:34.933475 20322 leader_election.cc:302] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 67b6992471b24ce0b726b833157df909, f283717ddaad49cd9d5fdfde6ef2d2fe, f6a302db476745678497bc364527197a; no voters:
I20260926 02:20:34.933771 20843 raft_consensus.cc:2781] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:34.933887 20843 raft_consensus.cc:482] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:34.933984 20843 raft_consensus.cc:3037] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:34.936244 20843 raft_consensus.cc:504] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.936906 20843 leader_election.cc:288] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 election: Requested vote from peers f6a302db476745678497bc364527197a (127.12.244.194:39225), 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613)
I20260926 02:20:34.937577 20536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c141e815df44afaa1f85af704977c08" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6a302db476745678497bc364527197a"
I20260926 02:20:34.937688 20679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c141e815df44afaa1f85af704977c08" candidate_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67b6992471b24ce0b726b833157df909"
I20260926 02:20:34.937882 20536 raft_consensus.cc:3037] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:34.937992 20679 raft_consensus.cc:3037] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:34.940137 20679 raft_consensus.cc:2454] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 1.
I20260926 02:20:34.940239 20536 raft_consensus.cc:2454] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f283717ddaad49cd9d5fdfde6ef2d2fe in term 1.
I20260926 02:20:34.940853 20323 leader_election.cc:302] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f283717ddaad49cd9d5fdfde6ef2d2fe, f6a302db476745678497bc364527197a; no voters:
I20260926 02:20:34.941228 20843 raft_consensus.cc:2781] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:34.941820 20843 raft_consensus.cc:686] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 LEADER]: Becoming Leader. State: Replica: f283717ddaad49cd9d5fdfde6ef2d2fe, State: Running, Role: LEADER
I20260926 02:20:34.942178 20843 consensus_queue.cc:237] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } }
I20260926 02:20:34.943266 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:34.942728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:50829)
0926 02:20:34.942920 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:34.943025 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:20:34.945516 20251 catalog_manager.cc:5495] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193). New cstate: current_term: 1 leader_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f283717ddaad49cd9d5fdfde6ef2d2fe" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 } health_report { overall_health: UNKNOWN } } }
I20260926 02:20:35.021938 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.016463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.12.244.193:45503)
0926 02:20:35.016669 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:35.016676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:35.016949 (+ 273us) server_negotiation.cc:407] Connection header received
0926 02:20:35.017092 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:35.017098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:35.017181 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:35.017290 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:35.017631 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.018733 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.020152 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.020811 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.020936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:35.021303 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:35.021477 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:35.021550 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:35.021670 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 02:20:35.021735 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:20:35.025303 20536 raft_consensus.cc:1261] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Refusing update from remote peer f283717ddaad49cd9d5fdfde6ef2d2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:35.025292 20679 raft_consensus.cc:1261] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Refusing update from remote peer f283717ddaad49cd9d5fdfde6ef2d2fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:35.026124 20846 consensus_queue.cc:1035] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "67b6992471b24ce0b726b833157df909" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:35.026516 20843 consensus_queue.cc:1035] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6a302db476745678497bc364527197a" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:35.029583 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.028800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:36111)
0926 02:20:35.029192 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.029327 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260926 02:20:35.030329 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.028848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:44309)
0926 02:20:35.029940 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.030089 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":917}
I20260926 02:20:35.043833 20853 mvcc.cc:205] Tried to move back new op lower bound from 7333434306656870400 to 7333434306325606400. Current Snapshot: MvccSnapshot[applied={T|T < 7333434306656870400}]
I20260926 02:20:35.044763 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.043590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:57667)
0926 02:20:35.044450 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.044560 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":747}
I20260926 02:20:35.046414 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.045746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.195:55735)
0926 02:20:35.046164 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.046261 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301}
I20260926 02:20:35.047544 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.046828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.194:40255)
0926 02:20:35.047163 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.047291 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 02:20:35.051250 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.050693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.12.244.193:49695)
0926 02:20:35.050954 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.051056 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 02:20:35.057293 20251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:35.057601 20251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:35.064271 20251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14b6519af9cc47d7a99b11fab42854b1: Sending DeleteTablet for 3 replicas of tablet 7c141e815df44afaa1f85af704977c08
I20260926 02:20:35.065389 20515 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c141e815df44afaa1f85af704977c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:35 UTC) from {username='oryx', principal='oryx/127.12.244.252@KRBTEST.COM'} at 127.0.0.1:42162
I20260926 02:20:35.065392 20367 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c141e815df44afaa1f85af704977c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:35 UTC) from {username='oryx', principal='oryx/127.12.244.252@KRBTEST.COM'} at 127.0.0.1:37104
I20260926 02:20:35.065384 20658 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c141e815df44afaa1f85af704977c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:35 UTC) from {username='oryx', principal='oryx/127.12.244.252@KRBTEST.COM'} at 127.0.0.1:54292
I20260926 02:20:35.067114 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 20229
I20260926 02:20:35.070541 20860 tablet_replica.cc:330] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: stopping tablet replica
I20260926 02:20:35.070778 20860 raft_consensus.cc:2229] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:35.071071 20860 raft_consensus.cc:2258] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:35.072463 20860 ts_tablet_manager.cc:1905] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:35.076725 20862 tablet_replica.cc:330] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: stopping tablet replica
I20260926 02:20:35.077193 20862 raft_consensus.cc:2229] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:35.077852 20862 raft_consensus.cc:2258] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:35.079375 20862 ts_tablet_manager.cc:1905] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:35.079932 20860 ts_tablet_manager.cc:1918] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:35.080097 20860 log.cc:1196] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/7c141e815df44afaa1f85af704977c08
I20260926 02:20:35.080588 20860 ts_tablet_manager.cc:1939] T 7c141e815df44afaa1f85af704977c08 P f6a302db476745678497bc364527197a: Deleting consensus metadata
I20260926 02:20:35.082185 20239 catalog_manager.cc:4841] TS f6a302db476745678497bc364527197a (127.12.244.194:39225): tablet 7c141e815df44afaa1f85af704977c08 (table test-table [id=c7974f41ebe348e080e46bbb99f5a74c]) successfully deleted
I20260926 02:20:35.082880 20861 tablet_replica.cc:330] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: stopping tablet replica
I20260926 02:20:35.083092 20861 raft_consensus.cc:2229] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:35.083395 20861 raft_consensus.cc:2258] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:35.084913 20861 ts_tablet_manager.cc:1905] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:35.089383 20862 ts_tablet_manager.cc:1918] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:35.089561 20862 log.cc:1196] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/7c141e815df44afaa1f85af704977c08
I20260926 02:20:35.090124 20862 ts_tablet_manager.cc:1939] T 7c141e815df44afaa1f85af704977c08 P f283717ddaad49cd9d5fdfde6ef2d2fe: Deleting consensus metadata
I20260926 02:20:35.091481 20238 catalog_manager.cc:4841] TS f283717ddaad49cd9d5fdfde6ef2d2fe (127.12.244.193:45503): tablet 7c141e815df44afaa1f85af704977c08 (table test-table [id=c7974f41ebe348e080e46bbb99f5a74c]) successfully deleted
I20260926 02:20:35.092224 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.068342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.12.244.193:45503)
0926 02:20:35.068532 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:35.068538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:35.070028 (+ 1490us) server_negotiation.cc:407] Connection header received
0926 02:20:35.070218 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:35.070224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:35.070354 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:35.070456 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:35.072893 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.074047 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.075951 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.076687 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.076796 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:35.080892 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:35.080916 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:35.080938 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:35.080987 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:35.087214 (+ 6227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:35.088586 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:35.088602 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:35.088608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:35.088888 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:35.089489 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:35.089496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:35.089499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:35.090151 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:35.090560 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:35.091444 (+ 884us) server_negotiation.cc:299] Negotiation successful
0926 02:20:35.091806 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:20:35.092558 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.084558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53314)
0926 02:20:35.092197 (+ 7639us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.092325 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7507,"thread_start_us":7406,"threads_started":1}
I20260926 02:20:35.093359 20861 ts_tablet_manager.cc:1918] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:35.093546 20861 log.cc:1196] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/7c141e815df44afaa1f85af704977c08
I20260926 02:20:35.093845 20428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:35.094084 20861 ts_tablet_manager.cc:1939] T 7c141e815df44afaa1f85af704977c08 P 67b6992471b24ce0b726b833157df909: Deleting consensus metadata
I20260926 02:20:35.094326 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.093704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53316)
0926 02:20:35.093972 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.094092 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 02:20:35.095327 20238 catalog_manager.cc:4841] TS 67b6992471b24ce0b726b833157df909 (127.12.244.195:36613): tablet 7c141e815df44afaa1f85af704977c08 (table test-table [id=c7974f41ebe348e080e46bbb99f5a74c]) successfully deleted
I20260926 02:20:35.225073 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 20229
I20260926 02:20:35.248729 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 20445
I20260926 02:20:35.264798 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.249641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.12.244.194:39225)
0926 02:20:35.250199 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:35.250204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:35.250221 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:20:35.250275 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:35.250279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:35.250359 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:35.250454 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:35.252052 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.252961 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.254475 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.254929 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.255044 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:35.258246 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:35.258270 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:35.258287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:35.258338 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:35.261465 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:35.262312 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:35.262322 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:35.262324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:35.262524 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:35.262966 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:35.262970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:35.262972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:35.263479 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:35.263711 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:35.264169 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 02:20:35.264443 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":308,"threads_started":1}
I20260926 02:20:35.265533 20577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:35.366060 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 20445
I20260926 02:20:35.384063 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 20595
I20260926 02:20:35.400626 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.384964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.12.244.195:36613)
0926 02:20:35.385561 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:35.385565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:35.385583 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:20:35.385656 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:35.385660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:35.385719 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:35.385819 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:35.387438 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.388173 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.389546 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:35.390181 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:35.390352 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:35.393433 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:35.393450 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:35.393468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:35.393527 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:35.396788 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:35.397811 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:35.397821 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:35.397823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:35.398070 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:35.398629 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:35.398633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:35.398635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:35.399234 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:35.399529 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:35.400071 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 02:20:35.400282 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
I20260926 02:20:35.401656 20719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:35.503693 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 20595
I20260926 02:20:35.521481 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 19919
I20260926 02:20:35.522830 20093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:35.619999 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.619044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53326)
0926 02:20:35.619734 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.619831 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":345,"threads_started":1}
I20260926 02:20:35.634378 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:35.633803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33863 (local address 127.0.0.1:53342)
0926 02:20:35.634061 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:35.634179 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:33863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 02:20:35.647437 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 19919
I20260926 02:20:35.670987 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 20018
I20260926 02:20:35.672269 20283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:35.798223 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 20018
2026-09-26T02:20:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12112 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:35 dist-test-slave-qfmw krb5kdc[20877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:35 dist-test-slave-qfmw krb5kdc[20877](info): set up 2 sockets
Sep 26 02:20:35 dist-test-slave-qfmw krb5kdc[20877](info): commencing operation
krb5kdc: starting...
W20260926 02:20:36.926632 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:37 dist-test-slave-qfmw krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab.
Entry for principal oryx/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab.
I20260926 02:20:37.163939 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:43003
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab
--principal=oryx/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:37.507438 20890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:37.507758 20890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:37.507929 20890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:37.515461 20890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:37.515586 20890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:20:37.515661 20890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:37.515763 20890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:37.515825 20890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:37.515911 20890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:37.526165 20890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:43003
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.254.keytab
--principal=oryx/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:37.527649 20890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:37.529073 20890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:37.542596 20895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:37.542626 20896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:37.542529 20898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:37.544054 20890 server_base.cc:1034] running on GCE node
I20260926 02:20:37.544966 20890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:37.546610 20890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:37.547869 20890 hybrid_clock.cc:648] HybridClock initialized: now 1790389237547765 us; error 64 us; skew 500 ppm
I20260926 02:20:37.548257 20890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:37 dist-test-slave-qfmw krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389237, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:37.551802 20890 init.cc:377] Logged in from keytab as oryx/127.12.244.254@KRBTEST.COM (short username oryx)
I20260926 02:20:37.553411 20890 webserver.cc:460] Webserver started at http://127.12.244.254:45339/ using document root <none> and password file <none>
I20260926 02:20:37.554131 20890 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:37.554221 20890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:37.554505 20890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:37.557595 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "9561fffef7a946e696e8d3b05ce62291"
format_stamp: "Formatted at 2026-09-26 02:20:37 on dist-test-slave-qfmw"
server_key: "79274fe1aaee132dc221e6a92e59c6c4"
server_key_iv: "06d7090353d0c4b5e183f0155ec6fac5"
server_key_version: "encryptionkey@0"
I20260926 02:20:37.558379 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "9561fffef7a946e696e8d3b05ce62291"
format_stamp: "Formatted at 2026-09-26 02:20:37 on dist-test-slave-qfmw"
server_key: "79274fe1aaee132dc221e6a92e59c6c4"
server_key_iv: "06d7090353d0c4b5e183f0155ec6fac5"
server_key_version: "encryptionkey@0"
I20260926 02:20:37.563812 20890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:20:37.567668 20905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:37.569316 20890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:37.569558 20890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "9561fffef7a946e696e8d3b05ce62291"
format_stamp: "Formatted at 2026-09-26 02:20:37 on dist-test-slave-qfmw"
server_key: "79274fe1aaee132dc221e6a92e59c6c4"
server_key_iv: "06d7090353d0c4b5e183f0155ec6fac5"
server_key_version: "encryptionkey@0"
I20260926 02:20:37.569808 20890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:37.582175 20890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:37.586908 20890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:37.587358 20890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:37.609167 20890 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:43003
I20260926 02:20:37.609185 20967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:43003 every 8 connection(s)
I20260926 02:20:37.610616 20890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:37.614790 20968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:37.616914 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 20890
I20260926 02:20:37.617116 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:37.617772 13267 external_mini_cluster.cc:1428] Setting key 530d65cb80c43907e80bcc830473ecee
I20260926 02:20:37.626564 20968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Bootstrap starting.
I20260926 02:20:37.630039 20968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:37.631152 20968 log.cc:824] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:20:37 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:37.634693 20968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: No bootstrap required, opened a new log
I20260926 02:20:37.643213 20971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:37.620005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.12.244.254:43003)
0926 02:20:37.620892 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:37.620906 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:37.620951 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:20:37.622678 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:37.622705 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:37.623236 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:37.623721 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:37.625224 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:37.626563 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:37.627991 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:37.628749 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:37.628972 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:37.633231 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:37.633243 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:37.633265 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:37.633313 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:37.637501 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:37.638343 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:37.638353 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:37.638364 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:37.638603 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:37.639522 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:37.639527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:37.639530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:37.640351 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:37.640661 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:37.641085 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 02:20:37.641617 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":430,"threads_started":1}
I20260926 02:20:37.645841 20968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561fffef7a946e696e8d3b05ce62291" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43003 } }
I20260926 02:20:37.646255 20968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:37.646402 20968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9561fffef7a946e696e8d3b05ce62291, State: Initialized, Role: FOLLOWER
I20260926 02:20:37.647245 20968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561fffef7a946e696e8d3b05ce62291" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43003 } }
I20260926 02:20:37.647572 20968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:37.647740 20968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:37.647972 20968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:37.650235 20968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561fffef7a946e696e8d3b05ce62291" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43003 } }
I20260926 02:20:37.650785 20968 leader_election.cc:302] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9561fffef7a946e696e8d3b05ce62291; no voters:
I20260926 02:20:37.651494 20968 leader_election.cc:288] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:37.651607 20973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:37.652917 20973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [term 1 LEADER]: Becoming Leader. State: Replica: 9561fffef7a946e696e8d3b05ce62291, State: Running, Role: LEADER
I20260926 02:20:37.653460 20973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561fffef7a946e696e8d3b05ce62291" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43003 } }
I20260926 02:20:37.653805 20968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:37.656175 20974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9561fffef7a946e696e8d3b05ce62291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561fffef7a946e696e8d3b05ce62291" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43003 } } }
I20260926 02:20:37.656502 20974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:37.656989 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9561fffef7a946e696e8d3b05ce62291. Latest consensus state: current_term: 1 leader_uuid: "9561fffef7a946e696e8d3b05ce62291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9561fffef7a946e696e8d3b05ce62291" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43003 } } }
I20260926 02:20:37.657266 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:37.657398 20980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:37.663542 20980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:37.664875 20990 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:20:37.673827 20980 catalog_manager.cc:1347] Generated new cluster ID: 7300b85b526342bbb686a36f04c3c7b4
I20260926 02:20:37.674072 20980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:37.690325 20980 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:37.691855 20980 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:37.698264 20980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Generated new TSK 0
I20260926 02:20:37.699450 20980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab.
Entry for principal oryx/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab.
I20260926 02:20:37.754971 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43003
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab
--principal=oryx/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:38.107646 20994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:38.107968 20994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:38.108067 20994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:38.116007 20994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:38.116195 20994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:38.116340 20994 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:20:38.116427 20994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:38.126689 20994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab
--principal=oryx/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:43003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:38.128106 20994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:38.129472 20994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:38.144299 20999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:38.144233 21000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:38.144198 21002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:38.145255 20994 server_base.cc:1034] running on GCE node
I20260926 02:20:38.146064 20994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:38.147243 20994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:38.148527 20994 hybrid_clock.cc:648] HybridClock initialized: now 1790389238148440 us; error 72 us; skew 500 ppm
I20260926 02:20:38.148885 20994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:38 dist-test-slave-qfmw krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:38.152426 20994 init.cc:377] Logged in from keytab as oryx/127.12.244.193@KRBTEST.COM (short username oryx)
I20260926 02:20:38.154145 20994 webserver.cc:460] Webserver started at http://127.12.244.193:43121/ using document root <none> and password file <none>
I20260926 02:20:38.154881 20994 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:38.155025 20994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:38.155412 20994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:38.158689 20994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "b2e6b851830a4d54afd303ef7702966a"
format_stamp: "Formatted at 2026-09-26 02:20:38 on dist-test-slave-qfmw"
server_key: "4c2f0a0bca62a3d7a0c131523e3262c3"
server_key_iv: "c2b4d3e0add4e9874b7756b8b98d09fb"
server_key_version: "encryptionkey@0"
I20260926 02:20:38.159648 20994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "b2e6b851830a4d54afd303ef7702966a"
format_stamp: "Formatted at 2026-09-26 02:20:38 on dist-test-slave-qfmw"
server_key: "4c2f0a0bca62a3d7a0c131523e3262c3"
server_key_iv: "c2b4d3e0add4e9874b7756b8b98d09fb"
server_key_version: "encryptionkey@0"
I20260926 02:20:38.165325 20994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:20:38.169466 21009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:38.171073 20994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:20:38.171339 20994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "b2e6b851830a4d54afd303ef7702966a"
format_stamp: "Formatted at 2026-09-26 02:20:38 on dist-test-slave-qfmw"
server_key: "4c2f0a0bca62a3d7a0c131523e3262c3"
server_key_iv: "c2b4d3e0add4e9874b7756b8b98d09fb"
server_key_version: "encryptionkey@0"
I20260926 02:20:38.171602 20994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:38.192179 20994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:38.197113 20994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:38.197465 20994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:38.200379 20994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:38.200501 20994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:38.200678 20994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:38.200795 20994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 02:20:38 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:38.226960 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.202109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:39889 (local address 127.12.244.254:43003)
0926 02:20:38.202608 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:38.202615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:38.202731 (+ 116us) server_negotiation.cc:407] Connection header received
0926 02:20:38.205446 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:38.205452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:38.205591 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:38.205724 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:38.208538 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.209635 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.212246 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.212898 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.213047 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.218792 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:38.218803 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:38.218811 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:38.218872 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:38.222550 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.223781 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.223787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.223790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.223952 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.224640 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.224648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.224654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.225077 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:38.225258 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:38.226073 (+ 815us) server_negotiation.cc:299] Negotiation successful
0926 02:20:38.226404 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":1020,"threads_started":1}
I20260926 02:20:38.227625 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.201629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43003 (local address 127.12.244.193:39889)
0926 02:20:38.202550 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:38.202601 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:38.204896 (+ 2295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:38.206200 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:38.206215 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:38.207193 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:38.208316 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.208351 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.209810 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.209821 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.212066 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.212088 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.213174 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.213182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.213588 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.214764 (+ 1176us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:38.214797 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:38.218355 (+ 3558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:38.222735 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.222752 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.222782 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.223570 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.224110 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.224115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.224119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.224470 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.225244 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:38.225251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:38.225641 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 02:20:38.225891 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:20:38.226317 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":573,"threads_started":1}
I20260926 02:20:38.249853 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.233508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43003 (local address 127.12.244.193:45929)
0926 02:20:38.234226 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:38.234242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:38.234384 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:38.234795 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:38.234798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:38.235333 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:38.235925 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.235945 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.237374 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.237382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.239210 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.239223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.240289 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.240300 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.240570 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.241451 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:38.241474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:38.242675 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:38.246630 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.246636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.246640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.247332 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.247829 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.247835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.247840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.248015 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.248870 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:38.248876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:38.249082 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 02:20:38.249198 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 02:20:38.249484 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":377,"threads_started":1}
I20260926 02:20:38.250278 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.233518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:45929 (local address 127.12.244.254:43003)
0926 02:20:38.233744 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:38.233753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:38.234326 (+ 573us) server_negotiation.cc:407] Connection header received
0926 02:20:38.234515 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:38.234523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:38.234631 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:38.234762 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:38.236170 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.237237 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.239456 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.240026 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.240177 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.242951 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:38.242959 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:38.242964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:38.243006 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:38.246469 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.247494 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.247500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.247504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.247673 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.248151 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.248156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.248159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.248609 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:38.248786 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:38.249416 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 02:20:38.249781 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:20:38.253460 20994 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:34521
I20260926 02:20:38.253501 21130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:34521 every 8 connection(s)
I20260926 02:20:38.255048 20994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:38.257643 21131 heartbeater.cc:347] Connected to a master server at 127.12.244.254:43003
I20260926 02:20:38.257899 21131 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:38.258647 21131 heartbeater.cc:511] Master 127.12.244.254:43003 requested a full tablet report, sending...
I20260926 02:20:38.260726 20922 ts_manager.cc:194] Registered new tserver with Master: b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521)
I20260926 02:20:38.262260 20922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:39889
I20260926 02:20:38.265007 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 20994
I20260926 02:20:38.265208 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:38.265887 13267 external_mini_cluster.cc:1428] Setting key 66052021e04889fd8aeb1b78141848e9
WARNING: no policy specified for oryx/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab.
Entry for principal oryx/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab.
I20260926 02:20:38.318324 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43003
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab
--principal=oryx/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:38.661243 21135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:38.661600 21135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:38.661780 21135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:38.670749 21135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:38.670913 21135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:38.671069 21135 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:20:38.671268 21135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:38.681733 21135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab
--principal=oryx/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:43003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:38.683200 21135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:38.684594 21135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:38.699553 21143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:38.699863 21141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:38.700809 21140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:38.702057 21135 server_base.cc:1034] running on GCE node
I20260926 02:20:38.702797 21135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:38.703894 21135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:38.705148 21135 hybrid_clock.cc:648] HybridClock initialized: now 1790389238705059 us; error 77 us; skew 500 ppm
I20260926 02:20:38.705502 21135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:38 dist-test-slave-qfmw krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:38.708943 21135 init.cc:377] Logged in from keytab as oryx/127.12.244.194@KRBTEST.COM (short username oryx)
I20260926 02:20:38.710603 21135 webserver.cc:460] Webserver started at http://127.12.244.194:44385/ using document root <none> and password file <none>
I20260926 02:20:38.711419 21135 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:38.711552 21135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:38.711882 21135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:38.714955 21135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "e8da33688e034d77b6cca969f18712d2"
format_stamp: "Formatted at 2026-09-26 02:20:38 on dist-test-slave-qfmw"
server_key: "1a291bfd80a2c82c94cbad98ef9565db"
server_key_iv: "27b0d267029e366acb54f18078eecb26"
server_key_version: "encryptionkey@0"
I20260926 02:20:38.715915 21135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "e8da33688e034d77b6cca969f18712d2"
format_stamp: "Formatted at 2026-09-26 02:20:38 on dist-test-slave-qfmw"
server_key: "1a291bfd80a2c82c94cbad98ef9565db"
server_key_iv: "27b0d267029e366acb54f18078eecb26"
server_key_version: "encryptionkey@0"
I20260926 02:20:38.721506 21135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:20:38.725356 21150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:38.726800 21135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:20:38.726996 21135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "e8da33688e034d77b6cca969f18712d2"
format_stamp: "Formatted at 2026-09-26 02:20:38 on dist-test-slave-qfmw"
server_key: "1a291bfd80a2c82c94cbad98ef9565db"
server_key_iv: "27b0d267029e366acb54f18078eecb26"
server_key_version: "encryptionkey@0"
I20260926 02:20:38.727284 21135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:38.747814 21135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:38.752611 21135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:38.752899 21135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:38.756067 21135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:38.756183 21135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:38.756340 21135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:38.756467 21135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 02:20:38 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389238, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:38.785059 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.756624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:56147 (local address 127.12.244.254:43003)
0926 02:20:38.759537 (+ 2913us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:38.759542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:38.759562 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:20:38.762060 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:38.762064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:38.762133 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:38.762244 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:38.765141 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.766224 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.769922 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.770360 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.770470 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.776831 (+ 6361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:38.776840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:38.776848 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:38.776901 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:38.780563 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.781875 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.781884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.781891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.782079 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.782614 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.782622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.782628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.783138 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:38.783338 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:38.784436 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0926 02:20:38.784659 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2809,"thread_start_us":2692,"threads_started":1}
I20260926 02:20:38.786875 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.757424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43003 (local address 127.12.244.194:56147)
0926 02:20:38.758526 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:38.758574 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:38.761667 (+ 3093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:38.762623 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:38.762641 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:38.763628 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:38.764871 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.764910 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.766433 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.766443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.769709 (+ 3266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.769732 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.770645 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.770655 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.771122 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.772042 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:38.772085 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:38.776482 (+ 4397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:38.780802 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.780813 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.780842 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.781684 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.782220 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.782227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.782232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.782476 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.783326 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:38.783339 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:38.783895 (+ 556us) client_negotiation.cc:769] Sending connection context
0926 02:20:38.784331 (+ 436us) client_negotiation.cc:240] Negotiation successful
0926 02:20:38.784940 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":421,"threads_started":1}
I20260926 02:20:38.810317 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.793290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:44349 (local address 127.12.244.254:43003)
0926 02:20:38.793531 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:38.793539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:38.794052 (+ 513us) server_negotiation.cc:407] Connection header received
0926 02:20:38.794214 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:38.794222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:38.794328 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:38.794469 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:38.795685 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.796974 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.799614 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.800088 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.800239 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.803211 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:38.803217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:38.803221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:38.803258 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:38.807253 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.807999 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.808007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.808012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.808182 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:38.808745 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:38.808753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:38.808758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:38.809130 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:38.809299 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:38.809722 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 02:20:38.809955 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:20:38.810565 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:38.793304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43003 (local address 127.12.244.194:44349)
0926 02:20:38.793943 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:38.793970 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:38.794088 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:38.794547 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:38.794550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:38.794992 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:38.795418 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.795429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.797177 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.797185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.799451 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:38.799468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:38.800326 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:38.800334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:38.800563 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:38.801422 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:38.801456 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:38.802879 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:38.807410 (+ 4531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.807416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.807421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.807846 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.808346 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:38.808353 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:38.808356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:38.808522 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:38.809256 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:38.809262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:38.809463 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 02:20:38.809608 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:20:38.810009 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":349,"threads_started":1}
I20260926 02:20:38.812168 21135 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:41017
I20260926 02:20:38.812204 21271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:41017 every 8 connection(s)
I20260926 02:20:38.813488 21135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:38.816262 21272 heartbeater.cc:347] Connected to a master server at 127.12.244.254:43003
I20260926 02:20:38.816504 21272 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:38.817175 21272 heartbeater.cc:511] Master 127.12.244.254:43003 requested a full tablet report, sending...
I20260926 02:20:38.818485 20922 ts_manager.cc:194] Registered new tserver with Master: e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017)
I20260926 02:20:38.819394 20922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:56147
I20260926 02:20:38.822010 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 21135
I20260926 02:20:38.822142 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:38.822521 13267 external_mini_cluster.cc:1428] Setting key 300331d7aa88e206bee187b2c5bf4ff1
WARNING: no policy specified for oryx/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab.
Entry for principal oryx/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab.
I20260926 02:20:38.871342 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:43003
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab
--principal=oryx/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:39.237515 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:39.237913 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:39.238116 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:39.246243 21276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:39.246423 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:39.246591 21276 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:20:39.246742 21276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:39.257264 21276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab
--principal=oryx/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.244.254:43003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:39.258771 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:39.260290 21276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 02:20:39.265596 21131 heartbeater.cc:503] Master 127.12.244.254:43003 was elected leader, sending a full tablet report...
W20260926 02:20:39.274619 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:39.274629 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:39.274719 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:39.276279 21276 server_base.cc:1034] running on GCE node
I20260926 02:20:39.277222 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:39.278445 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:39.279806 21276 hybrid_clock.cc:648] HybridClock initialized: now 1790389239279649 us; error 50 us; skew 500 ppm
I20260926 02:20:39.280193 21276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:39 dist-test-slave-qfmw krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389239, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:39.284049 21276 init.cc:377] Logged in from keytab as oryx/127.12.244.195@KRBTEST.COM (short username oryx)
I20260926 02:20:39.285892 21276 webserver.cc:460] Webserver started at http://127.12.244.195:37363/ using document root <none> and password file <none>
I20260926 02:20:39.286705 21276 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:39.286805 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:39.287278 21276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:39.290588 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02"
format_stamp: "Formatted at 2026-09-26 02:20:39 on dist-test-slave-qfmw"
server_key: "0ddafa2a518234ac4ce400abeb3bc687"
server_key_iv: "950a03c931a80cf236908fcab58ca3dc"
server_key_version: "encryptionkey@0"
I20260926 02:20:39.291524 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02"
format_stamp: "Formatted at 2026-09-26 02:20:39 on dist-test-slave-qfmw"
server_key: "0ddafa2a518234ac4ce400abeb3bc687"
server_key_iv: "950a03c931a80cf236908fcab58ca3dc"
server_key_version: "encryptionkey@0"
I20260926 02:20:39.297443 21276 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 02:20:39.301429 21291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:39.302965 21276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:20:39.303200 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02"
format_stamp: "Formatted at 2026-09-26 02:20:39 on dist-test-slave-qfmw"
server_key: "0ddafa2a518234ac4ce400abeb3bc687"
server_key_iv: "950a03c931a80cf236908fcab58ca3dc"
server_key_version: "encryptionkey@0"
I20260926 02:20:39.303453 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:39.320370 21276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:39.325008 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:39.325306 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:39.328191 21276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:39.328279 21276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:39.328461 21276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:39.328519 21276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:20:39 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389239, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:39.355440 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.328913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:52827 (local address 127.12.244.254:43003)
0926 02:20:39.329616 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.329646 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.330868 (+ 1222us) server_negotiation.cc:407] Connection header received
0926 02:20:39.333278 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.333284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.333397 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.333531 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:39.336238 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.337343 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.340914 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.341381 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.341498 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.347185 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:39.347192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:39.347195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:39.347236 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:39.351293 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.352243 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.352252 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.352258 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.352481 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.352965 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.352972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.352978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.353467 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:39.353645 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.354527 (+ 882us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.354835 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":428,"threads_started":1}
I20260926 02:20:39.356472 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.329680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43003 (local address 127.12.244.195:52827)
0926 02:20:39.330649 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.330700 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.332952 (+ 2252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.333833 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.333848 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.334879 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:39.335998 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.336039 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.337549 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.337556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.340643 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.340667 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.341675 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.341687 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.342107 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.343160 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:39.343193 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:39.346812 (+ 3619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:39.351534 (+ 4722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:39.351542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:39.351559 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:39.352061 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:39.352621 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:39.352624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:39.352627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:39.352829 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:39.353611 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:39.353618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:39.354003 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.354367 (+ 364us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.354931 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":389,"threads_started":1}
I20260926 02:20:39.378549 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.362833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:39079 (local address 127.12.244.254:43003)
0926 02:20:39.363136 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.363144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.364027 (+ 883us) server_negotiation.cc:407] Connection header received
0926 02:20:39.364241 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.364253 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.364328 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.364439 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:39.365485 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.366220 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.367807 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.368634 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.368815 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.371414 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:39.371421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:39.371426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:39.371468 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:39.375279 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.376399 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.376403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.376406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.376548 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.377033 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.377039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.377042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.377349 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:39.377519 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.377947 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.378174 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:20:39.378648 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.362809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43003 (local address 127.12.244.195:39079)
0926 02:20:39.363910 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.363927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.364103 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.364466 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.364470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.365033 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:39.365325 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.365335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.366375 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.366381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.367627 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.367650 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.368858 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.368866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.369088 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.369811 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:39.369836 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:39.371204 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:39.375471 (+ 4267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:39.375477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:39.375482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:39.376225 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:39.376670 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:39.376678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:39.376683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:39.376894 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:39.377475 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:39.377480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:39.377673 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.377820 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.378175 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":732,"threads_started":1}
I20260926 02:20:39.386600 21276 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:44615
I20260926 02:20:39.386633 21411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:44615 every 8 connection(s)
I20260926 02:20:39.387907 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:39.390204 21412 heartbeater.cc:347] Connected to a master server at 127.12.244.254:43003
I20260926 02:20:39.390445 21412 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:39.391197 21412 heartbeater.cc:511] Master 127.12.244.254:43003 requested a full tablet report, sending...
I20260926 02:20:39.392711 20922 ts_manager.cc:194] Registered new tserver with Master: cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615)
I20260926 02:20:39.393496 20922 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:52827
I20260926 02:20:39.395716 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 21276
I20260926 02:20:39.395877 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:39.396296 13267 external_mini_cluster.cc:1428] Setting key 27f0d0007ba81e8666ce2a81c111ecad
I20260926 02:20:39.400043 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:20:39 dist-test-slave-qfmw krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389239, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:20:39 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:39 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:20:39.456616 21300 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:39.442153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.12.244.254:43003)
0926 02:20:39.442324 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.442330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.442610 (+ 280us) server_negotiation.cc:407] Connection header received
0926 02:20:39.442762 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.442766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.442918 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.443040 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:39.444896 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.446069 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.448350 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.449071 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.449221 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.456367 (+ 7146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33002: BlockingRecv error: recv got EOF from 127.0.0.1:33002 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 02:20:39 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389239, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:39.473037 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.459953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.12.244.254:43003)
0926 02:20:39.460201 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.460207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.460408 (+ 201us) server_negotiation.cc:407] Connection header received
0926 02:20:39.460553 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.460557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.460618 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.460699 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:39.461797 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.462501 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.463791 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.464235 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.464338 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.468105 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:39.468111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:39.468114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:39.468149 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:39.470597 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.471303 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.471307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.471310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.471445 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.471814 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.471816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.471818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.472068 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:39.472250 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.472535 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.472704 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:20:39.477593 20922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:20:39.484555 20922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:39.522256 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.498143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.12.244.195:44615)
0926 02:20:39.513473 (+ 15330us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.513481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.513513 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:20:39.513592 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.513598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.513781 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.514031 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:39.515426 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.517141 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.519930 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.521023 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.521242 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.521452 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.521706 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.521874 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15130,"thread_start_us":14932,"threads_started":1}
I20260926 02:20:39.522370 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.497559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44615 (local address 127.0.0.1:43190)
0926 02:20:39.510102 (+ 12543us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.510127 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.510362 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.514339 (+ 3977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.514347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.514387 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:39.515127 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.515144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.517368 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.517372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.519747 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.519761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.521234 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.521240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.521438 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.521495 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.521653 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.521874 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12416,"mutex_wait_us":114,"spinlock_wait_cycles":13056,"thread_start_us":12306,"threads_started":1}
I20260926 02:20:39.522958 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.497464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:34521 (local address 127.0.0.1:41022)
0926 02:20:39.510065 (+ 12601us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.510127 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.510362 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.514463 (+ 4101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.514468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.514492 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:39.515375 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.515386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.517312 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.517317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.520007 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.520021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.522105 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.522111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.522287 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.522323 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.522405 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.522545 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12193,"mutex_wait_us":185,"spinlock_wait_cycles":2176,"thread_start_us":13040,"threads_started":1}
I20260926 02:20:39.523442 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.498512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.12.244.194:41017)
0926 02:20:39.512450 (+ 13938us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.512465 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.513336 (+ 871us) server_negotiation.cc:407] Connection header received
0926 02:20:39.513471 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.513481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.513780 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.514036 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:39.515407 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.516377 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.519772 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.521208 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.521401 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.521574 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.522945 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.523180 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13756,"thread_start_us":13637,"threads_started":1}
I20260926 02:20:39.523733 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.497670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.12.244.193:34521)
0926 02:20:39.513852 (+ 16182us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.513858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.513884 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:20:39.514002 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.514008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.514251 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.514471 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:39.515584 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.517139 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.520189 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.521642 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.521811 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.521959 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.523278 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.523458 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16009,"thread_start_us":15879,"threads_started":1}
I20260926 02:20:39.524024 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.498008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:41017 (local address 127.0.0.1:58266)
0926 02:20:39.512956 (+ 14948us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.512981 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.513176 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.514339 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.514347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.514390 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:39.515182 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.515197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.516520 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.516528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.519582 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.519599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.521738 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.521746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.521903 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.521932 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.522012 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.522164 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14811,"mutex_wait_us":229,"spinlock_wait_cycles":2304,"thread_start_us":14679,"threads_started":1}
I20260926 02:20:39.526883 21342 tablet_service.cc:1467] Processing CreateTablet for tablet 5b973e18393d48d799456d210c1f9a72 (DEFAULT_TABLE table=test-table [id=ec303f513243499f80e5e4241480f00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:39.527665 21060 tablet_service.cc:1467] Processing CreateTablet for tablet 5b973e18393d48d799456d210c1f9a72 (DEFAULT_TABLE table=test-table [id=ec303f513243499f80e5e4241480f00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:39.527657 21201 tablet_service.cc:1467] Processing CreateTablet for tablet 5b973e18393d48d799456d210c1f9a72 (DEFAULT_TABLE table=test-table [id=ec303f513243499f80e5e4241480f00a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:39.528913 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b973e18393d48d799456d210c1f9a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.528995 21060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b973e18393d48d799456d210c1f9a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.529036 21201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b973e18393d48d799456d210c1f9a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.542248 21436 tablet_bootstrap.cc:492] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Bootstrap starting.
I20260926 02:20:39.545954 21438 tablet_bootstrap.cc:492] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Bootstrap starting.
I20260926 02:20:39.546658 21436 tablet_bootstrap.cc:654] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.547549 21437 tablet_bootstrap.cc:492] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Bootstrap starting.
I20260926 02:20:39.548184 21436 log.cc:824] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:39.549582 21438 tablet_bootstrap.cc:654] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.550926 21438 log.cc:824] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:39.550983 21437 tablet_bootstrap.cc:654] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.551913 21436 tablet_bootstrap.cc:492] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: No bootstrap required, opened a new log
I20260926 02:20:39.552204 21436 ts_tablet_manager.cc:1397] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 02:20:39.552384 21437 log.cc:824] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:39.555351 21438 tablet_bootstrap.cc:492] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: No bootstrap required, opened a new log
I20260926 02:20:39.555586 21437 tablet_bootstrap.cc:492] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: No bootstrap required, opened a new log
I20260926 02:20:39.555634 21438 ts_tablet_manager.cc:1397] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:20:39.555907 21437 ts_tablet_manager.cc:1397] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:20:39.563596 21436 raft_consensus.cc:348] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.564123 21436 raft_consensus.cc:374] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:39.564221 21436 raft_consensus.cc:729] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfb3f232abc4c03af03ec3e6dfc6c02, State: Initialized, Role: FOLLOWER
I20260926 02:20:39.564198 21437 raft_consensus.cc:348] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.564658 21437 raft_consensus.cc:374] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:39.564774 21437 raft_consensus.cc:729] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Initialized, Role: FOLLOWER
I20260926 02:20:39.564915 21436 consensus_queue.cc:260] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.565447 21437 consensus_queue.cc:260] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.566236 21412 heartbeater.cc:503] Master 127.12.244.254:43003 was elected leader, sending a full tablet report...
I20260926 02:20:39.567020 21436 ts_tablet_manager.cc:1428] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260926 02:20:39.568504 21437 ts_tablet_manager.cc:1428] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260926 02:20:39.569412 21438 raft_consensus.cc:348] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.569932 21438 raft_consensus.cc:374] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:39.570078 21438 raft_consensus.cc:729] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8da33688e034d77b6cca969f18712d2, State: Initialized, Role: FOLLOWER
I20260926 02:20:39.570861 21438 consensus_queue.cc:260] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.572010 21272 heartbeater.cc:503] Master 127.12.244.254:43003 was elected leader, sending a full tablet report...
I20260926 02:20:39.572906 21438 ts_tablet_manager.cc:1428] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260926 02:20:39.639585 21413 tablet.cc:2369] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:20:39.642370 21444 raft_consensus.cc:482] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:39.642657 21444 raft_consensus.cc:504] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.644472 21444 leader_election.cc:288] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521), cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615)
I20260926 02:20:39.654727 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.645099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:57827 (local address 127.12.244.193:34521)
0926 02:20:39.645338 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.645344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.646060 (+ 716us) server_negotiation.cc:407] Connection header received
0926 02:20:39.646280 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.646285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.646371 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.646481 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:39.647459 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.648585 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.651940 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.653281 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.653463 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.653543 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.654277 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.654444 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:20:39.654737 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.645018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:46173 (local address 127.12.244.195:44615)
0926 02:20:39.645220 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.645227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.646213 (+ 986us) server_negotiation.cc:407] Connection header received
0926 02:20:39.646289 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.646294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.646371 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.646489 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:39.647457 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.648611 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.651940 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.653263 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.653463 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.653543 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.654277 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.654445 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:20:39.654917 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.645209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44615 (local address 127.12.244.194:46173)
0926 02:20:39.646055 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.646070 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.646169 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.646535 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.646540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.646596 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:39.647188 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.647206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.648750 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.648756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.651690 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.651711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.653638 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.653645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.653893 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.654056 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.654170 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.654393 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":377,"spinlock_wait_cycles":6144,"thread_start_us":543,"threads_started":1}
I20260926 02:20:39.654983 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.644989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:34521 (local address 127.12.244.194:57827)
0926 02:20:39.645911 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.645933 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.646115 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.646535 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.646540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.646595 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:39.647262 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.647273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.648751 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.648756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.651690 (+ 2934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.651711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.653643 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.653648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.653975 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.654059 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.654170 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.654393 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":292,"spinlock_wait_cycles":33536,"thread_start_us":445,"threads_started":1}
I20260926 02:20:39.656188 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b973e18393d48d799456d210c1f9a72" candidate_uuid: "e8da33688e034d77b6cca969f18712d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" is_pre_election: true
I20260926 02:20:39.656206 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b973e18393d48d799456d210c1f9a72" candidate_uuid: "e8da33688e034d77b6cca969f18712d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e6b851830a4d54afd303ef7702966a" is_pre_election: true
I20260926 02:20:39.656759 21367 raft_consensus.cc:2454] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8da33688e034d77b6cca969f18712d2 in term 0.
I20260926 02:20:39.656948 21080 raft_consensus.cc:2454] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8da33688e034d77b6cca969f18712d2 in term 0.
I20260926 02:20:39.657567 21154 leader_election.cc:302] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdfb3f232abc4c03af03ec3e6dfc6c02, e8da33688e034d77b6cca969f18712d2; no voters:
I20260926 02:20:39.658018 21444 raft_consensus.cc:2781] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:39.658188 21444 raft_consensus.cc:482] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:39.658293 21444 raft_consensus.cc:3037] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:39.660753 21444 raft_consensus.cc:504] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.661342 21444 leader_election.cc:288] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [CANDIDATE]: Term 1 election: Requested vote from peers b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521), cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615)
I20260926 02:20:39.661859 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b973e18393d48d799456d210c1f9a72" candidate_uuid: "e8da33688e034d77b6cca969f18712d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02"
I20260926 02:20:39.662058 21367 raft_consensus.cc:3037] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:39.664381 21367 raft_consensus.cc:2454] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8da33688e034d77b6cca969f18712d2 in term 1.
I20260926 02:20:39.664886 21154 leader_election.cc:302] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdfb3f232abc4c03af03ec3e6dfc6c02, e8da33688e034d77b6cca969f18712d2; no voters:
I20260926 02:20:39.664870 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b973e18393d48d799456d210c1f9a72" candidate_uuid: "e8da33688e034d77b6cca969f18712d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e6b851830a4d54afd303ef7702966a"
I20260926 02:20:39.665105 21080 raft_consensus.cc:3037] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:39.665201 21444 raft_consensus.cc:2781] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:39.665725 21444 raft_consensus.cc:686] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 1 LEADER]: Becoming Leader. State: Replica: e8da33688e034d77b6cca969f18712d2, State: Running, Role: LEADER
I20260926 02:20:39.666324 21444 consensus_queue.cc:237] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:39.668479 21080 raft_consensus.cc:2454] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8da33688e034d77b6cca969f18712d2 in term 1.
I20260926 02:20:39.672252 20922 catalog_manager.cc:5495] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8da33688e034d77b6cca969f18712d2 (127.12.244.194). New cstate: current_term: 1 leader_uuid: "e8da33688e034d77b6cca969f18712d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } health_report { overall_health: HEALTHY } } }
I20260926 02:20:39.718487 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.712992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.12.244.194:41017)
0926 02:20:39.713193 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.713197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.713215 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:20:39.713374 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.713378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.713459 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.713602 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:39.713990 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.714750 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.716228 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.716649 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.716770 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.717305 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:39.717947 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:39.718065 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.718193 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.718270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:20:39.727212 21367 raft_consensus.cc:1261] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Refusing update from remote peer e8da33688e034d77b6cca969f18712d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:39.727203 21080 raft_consensus.cc:1261] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Refusing update from remote peer e8da33688e034d77b6cca969f18712d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:39.728303 21447 consensus_queue.cc:1035] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:39.728695 21444 consensus_queue.cc:1035] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:39.757550 20922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:39.757848 20922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:39.761829 20922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Sending DeleteTablet for 3 replicas of tablet 5b973e18393d48d799456d210c1f9a72
I20260926 02:20:39.763078 21342 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b973e18393d48d799456d210c1f9a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:39 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:43190
I20260926 02:20:39.763078 21060 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b973e18393d48d799456d210c1f9a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:39 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41022
I20260926 02:20:39.763219 21201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b973e18393d48d799456d210c1f9a72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:39 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:58266
I20260926 02:20:39.764231 21462 tablet_replica.cc:330] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: stopping tablet replica
I20260926 02:20:39.764704 21462 raft_consensus.cc:2229] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:39.765422 21462 raft_consensus.cc:2258] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:39.765717 20922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:20:39.766618 20922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:39.767583 21462 ts_tablet_manager.cc:1905] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:39.775373 21462 ts_tablet_manager.cc:1918] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:39.775573 21462 log.cc:1196] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/5b973e18393d48d799456d210c1f9a72
I20260926 02:20:39.775790 21460 tablet_replica.cc:330] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: stopping tablet replica
I20260926 02:20:39.776247 21462 ts_tablet_manager.cc:1939] T 5b973e18393d48d799456d210c1f9a72 P e8da33688e034d77b6cca969f18712d2: Deleting consensus metadata
I20260926 02:20:39.776698 21459 tablet_replica.cc:330] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: stopping tablet replica
I20260926 02:20:39.777115 21459 raft_consensus.cc:2229] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:39.777611 21459 raft_consensus.cc:2258] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:39.776295 21460 raft_consensus.cc:2229] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:39.778373 21460 raft_consensus.cc:2258] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:39.778594 20907 catalog_manager.cc:4841] TS e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017): tablet 5b973e18393d48d799456d210c1f9a72 (table test-table [id=ec303f513243499f80e5e4241480f00a]) successfully deleted
I20260926 02:20:39.779023 21201 tablet_service.cc:1467] Processing CreateTablet for tablet ebbb474a921e471f95288126eceea84e (DEFAULT_TABLE table=test-table [id=4a8d5379dea74e379cd8807092a50b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:39.779522 21201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbb474a921e471f95288126eceea84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.779678 21059 tablet_service.cc:1467] Processing CreateTablet for tablet ebbb474a921e471f95288126eceea84e (DEFAULT_TABLE table=test-table [id=4a8d5379dea74e379cd8807092a50b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:39.780083 21059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbb474a921e471f95288126eceea84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.780546 21460 ts_tablet_manager.cc:1905] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:39.784063 21342 tablet_service.cc:1467] Processing CreateTablet for tablet ebbb474a921e471f95288126eceea84e (DEFAULT_TABLE table=test-table [id=4a8d5379dea74e379cd8807092a50b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:39.784518 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebbb474a921e471f95288126eceea84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.786381 21438 tablet_bootstrap.cc:492] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Bootstrap starting.
I20260926 02:20:39.788156 21438 tablet_bootstrap.cc:654] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.791191 21437 tablet_bootstrap.cc:492] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Bootstrap starting.
I20260926 02:20:39.791692 21459 ts_tablet_manager.cc:1905] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:39.792848 21437 tablet_bootstrap.cc:654] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.795008 21437 tablet_bootstrap.cc:492] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: No bootstrap required, opened a new log
I20260926 02:20:39.795249 21437 ts_tablet_manager.cc:1397] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:39.795341 21436 tablet_bootstrap.cc:492] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: Bootstrap starting.
I20260926 02:20:39.796972 21436 tablet_bootstrap.cc:654] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.797605 21437 raft_consensus.cc:348] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.797992 21437 raft_consensus.cc:374] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:39.798161 21437 raft_consensus.cc:729] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Initialized, Role: FOLLOWER
I20260926 02:20:39.798609 21437 consensus_queue.cc:260] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.799649 21437 ts_tablet_manager.cc:1428] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 02:20:39.803040 21438 tablet_bootstrap.cc:492] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: No bootstrap required, opened a new log
I20260926 02:20:39.803236 21438 ts_tablet_manager.cc:1397] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20260926 02:20:39.804108 21438 raft_consensus.cc:348] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.804344 21438 raft_consensus.cc:374] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:39.804446 21438 raft_consensus.cc:729] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8da33688e034d77b6cca969f18712d2, State: Initialized, Role: FOLLOWER
I20260926 02:20:39.804739 21438 consensus_queue.cc:260] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.807448 21438 ts_tablet_manager.cc:1428] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 02:20:39.813416 21436 tablet_bootstrap.cc:492] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: No bootstrap required, opened a new log
I20260926 02:20:39.813591 21436 ts_tablet_manager.cc:1397] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.004s
I20260926 02:20:39.814404 21436 raft_consensus.cc:348] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.814617 21436 raft_consensus.cc:374] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:39.814698 21436 raft_consensus.cc:729] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfb3f232abc4c03af03ec3e6dfc6c02, State: Initialized, Role: FOLLOWER
I20260926 02:20:39.815019 21436 consensus_queue.cc:260] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
W20260926 02:20:39.816171 21273 tablet.cc:2369] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:20:39.817329 21436 ts_tablet_manager.cc:1428] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:39.819458 21459 ts_tablet_manager.cc:1918] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:39.819747 21459 log.cc:1196] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/5b973e18393d48d799456d210c1f9a72
I20260926 02:20:39.820380 21459 ts_tablet_manager.cc:1939] T 5b973e18393d48d799456d210c1f9a72 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting consensus metadata
I20260926 02:20:39.820616 21460 ts_tablet_manager.cc:1918] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:39.820813 21460 log.cc:1196] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/5b973e18393d48d799456d210c1f9a72
I20260926 02:20:39.821342 21460 ts_tablet_manager.cc:1939] T 5b973e18393d48d799456d210c1f9a72 P b2e6b851830a4d54afd303ef7702966a: Deleting consensus metadata
I20260926 02:20:39.823571 20908 catalog_manager.cc:4841] TS b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521): tablet 5b973e18393d48d799456d210c1f9a72 (table test-table [id=ec303f513243499f80e5e4241480f00a]) successfully deleted
I20260926 02:20:39.824959 20909 catalog_manager.cc:4841] TS cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615): tablet 5b973e18393d48d799456d210c1f9a72 (table test-table [id=ec303f513243499f80e5e4241480f00a]) successfully deleted
I20260926 02:20:39.869470 21442 raft_consensus.cc:482] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:39.869760 21442 raft_consensus.cc:504] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.871579 21442 leader_election.cc:288] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521), e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017)
I20260926 02:20:39.881850 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.872468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:33407 (local address 127.12.244.193:34521)
0926 02:20:39.872661 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.872668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.872688 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:20:39.872908 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.872915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.873000 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.873117 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:39.875001 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.876143 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.879531 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.880831 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.881020 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.881101 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.881416 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.881558 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:20:39.881930 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.871813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:34521 (local address 127.12.244.195:33407)
0926 02:20:39.872519 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.872543 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.872710 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.873347 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.873353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.873392 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:39.874795 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.874813 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.876329 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.876342 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.879319 (+ 2977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.879336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.881052 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.881058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.881210 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.881238 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.881373 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.881534 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":87,"thread_start_us":543,"threads_started":1}
I20260926 02:20:39.883193 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbb474a921e471f95288126eceea84e" candidate_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e6b851830a4d54afd303ef7702966a" is_pre_election: true
I20260926 02:20:39.883483 21080 raft_consensus.cc:2454] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdfb3f232abc4c03af03ec3e6dfc6c02 in term 0.
I20260926 02:20:39.883802 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.872045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:35651 (local address 127.12.244.194:41017)
0926 02:20:39.872263 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.872269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.873452 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 02:20:39.873749 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.873755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.873842 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.873938 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:39.875223 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.876443 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.881210 (+ 4767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.882514 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.882673 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.882749 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.883192 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.883342 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:20:39.883837 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.871986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:41017 (local address 127.12.244.195:35651)
0926 02:20:39.873348 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.873366 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.873569 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.873953 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.873958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.873985 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:39.875041 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.875058 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.876592 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.876598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.879316 (+ 2718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.879336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.882738 (+ 3402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.882749 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.882971 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.883010 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.883170 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.883327 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":115,"thread_start_us":1016,"threads_started":1}
I20260926 02:20:39.884075 21294 leader_election.cc:302] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, cdfb3f232abc4c03af03ec3e6dfc6c02; no voters:
I20260926 02:20:39.884599 21442 raft_consensus.cc:2781] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:39.884758 21442 raft_consensus.cc:482] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:39.884866 21442 raft_consensus.cc:3037] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:39.885120 21221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbb474a921e471f95288126eceea84e" candidate_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2" is_pre_election: true
I20260926 02:20:39.885581 21221 raft_consensus.cc:2454] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdfb3f232abc4c03af03ec3e6dfc6c02 in term 0.
I20260926 02:20:39.887566 21442 raft_consensus.cc:504] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.888201 21442 leader_election.cc:288] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [CANDIDATE]: Term 1 election: Requested vote from peers b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521), e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017)
I20260926 02:20:39.888767 21080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbb474a921e471f95288126eceea84e" candidate_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e6b851830a4d54afd303ef7702966a"
I20260926 02:20:39.888775 21221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebbb474a921e471f95288126eceea84e" candidate_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2"
I20260926 02:20:39.888985 21221 raft_consensus.cc:3037] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:39.888990 21080 raft_consensus.cc:3037] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:39.891557 21080 raft_consensus.cc:2454] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdfb3f232abc4c03af03ec3e6dfc6c02 in term 1.
I20260926 02:20:39.892038 21294 leader_election.cc:302] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, cdfb3f232abc4c03af03ec3e6dfc6c02; no voters:
I20260926 02:20:39.892385 21442 raft_consensus.cc:2781] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:39.892349 21221 raft_consensus.cc:2454] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdfb3f232abc4c03af03ec3e6dfc6c02 in term 1.
I20260926 02:20:39.892890 21442 raft_consensus.cc:686] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 LEADER]: Becoming Leader. State: Replica: cdfb3f232abc4c03af03ec3e6dfc6c02, State: Running, Role: LEADER
I20260926 02:20:39.893667 21442 consensus_queue.cc:237] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:39.897873 20921 catalog_manager.cc:5495] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } health_report { overall_health: HEALTHY } } }
Sep 26 02:20:39 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389237, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.254@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:39.971143 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.949759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:43003 (local address 127.0.0.1:33022)
0926 02:20:39.950479 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:39.950495 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:39.950623 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:39.950996 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:39.950999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:39.951798 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:39.952056 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.952066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.953597 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.953601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.954615 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:39.954632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.955604 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.955609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:39.955721 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.962151 (+ 6430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:39.962174 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:39.964533 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:39.968480 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:39.968494 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:39.968500 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:39.968962 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:39.969354 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:39.969357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:39.969360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:39.969494 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:39.970150 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:39.970158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:39.970321 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 02:20:39.970475 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 02:20:39.970750 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":396,"threads_started":1}
I20260926 02:20:39.971367 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:39.949740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.12.244.254:43003)
0926 02:20:39.950009 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:39.950016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:39.950584 (+ 568us) server_negotiation.cc:407] Connection header received
0926 02:20:39.950740 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:39.950745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:39.950835 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:39.950956 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:39.952222 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.953384 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.954760 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:39.955421 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:39.955553 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:39.964776 (+ 9223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:39.964784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:39.964799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:39.964843 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:39.968286 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.969089 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.969095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.969099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.969236 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:39.969652 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:39.969657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:39.969660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:39.970005 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:39.970142 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:39.970600 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 02:20:39.970928 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":123}
I20260926 02:20:39.977120 20921 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:33022:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:20:39.978230 20921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:39.985246 21059 tablet_service.cc:1467] Processing CreateTablet for tablet f9c3516db0f543d888137ae9f41b7848 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4664d5072bad449ba4c5d1ba96a86035]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:20:39.985847 21059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c3516db0f543d888137ae9f41b7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.985683 21201 tablet_service.cc:1467] Processing CreateTablet for tablet f9c3516db0f543d888137ae9f41b7848 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4664d5072bad449ba4c5d1ba96a86035]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:20:39.986227 21201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c3516db0f543d888137ae9f41b7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.986449 21342 tablet_service.cc:1467] Processing CreateTablet for tablet f9c3516db0f543d888137ae9f41b7848 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4664d5072bad449ba4c5d1ba96a86035]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:20:39.986932 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c3516db0f543d888137ae9f41b7848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:39.993543 21437 tablet_bootstrap.cc:492] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: Bootstrap starting.
I20260926 02:20:39.993562 21438 tablet_bootstrap.cc:492] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: Bootstrap starting.
I20260926 02:20:39.995208 21436 tablet_bootstrap.cc:492] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: Bootstrap starting.
I20260926 02:20:39.995332 21437 tablet_bootstrap.cc:654] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.995345 21438 tablet_bootstrap.cc:654] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:39.996840 21436 tablet_bootstrap.cc:654] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:40.005470 21436 tablet_bootstrap.cc:492] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: No bootstrap required, opened a new log
I20260926 02:20:40.005728 21436 ts_tablet_manager.cc:1397] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 02:20:40.005923 21437 tablet_bootstrap.cc:492] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: No bootstrap required, opened a new log
I20260926 02:20:40.006119 21437 ts_tablet_manager.cc:1397] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260926 02:20:40.006728 21436 raft_consensus.cc:348] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.007050 21436 raft_consensus.cc:374] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.006965 21437 raft_consensus.cc:348] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.007172 21436 raft_consensus.cc:729] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfb3f232abc4c03af03ec3e6dfc6c02, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.007256 21437 raft_consensus.cc:374] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.007339 21437 raft_consensus.cc:729] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.007637 21437 consensus_queue.cc:260] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.007670 21436 consensus_queue.cc:260] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.008163 21442 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.008180 21443 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.008518 21443 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:20:40.008528 21442 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 02:20:40.009339 21132 tablet.cc:2369] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:20:40.009567 21438 tablet_bootstrap.cc:492] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: No bootstrap required, opened a new log
I20260926 02:20:40.009987 21438 ts_tablet_manager.cc:1397] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260926 02:20:40.010397 21437 ts_tablet_manager.cc:1428] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:40.011052 21438 raft_consensus.cc:348] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.011374 21438 raft_consensus.cc:374] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.011482 21438 raft_consensus.cc:729] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8da33688e034d77b6cca969f18712d2, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.011786 21438 consensus_queue.cc:260] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.012197 21436 ts_tablet_manager.cc:1428] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 02:20:40.012279 21447 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.012660 21447 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:20:40.014430 21438 ts_tablet_manager.cc:1428] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 02:20:40.021965 21443 raft_consensus.cc:482] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:40.022157 21443 raft_consensus.cc:504] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.023887 21443 leader_election.cc:288] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017), cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615)
I20260926 02:20:40.031435 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.024931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:43539 (local address 127.12.244.195:44615)
0926 02:20:40.025110 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.025115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.025135 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:20:40.025322 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.025328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.025414 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.025467 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:40.026138 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.026878 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.029233 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.030365 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.030517 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.030617 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.030992 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.031149 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:20:40.031589 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.024240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44615 (local address 127.12.244.193:43539)
0926 02:20:40.024981 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:40.025008 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:40.025153 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:40.025634 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:40.025637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:40.025657 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:40.025980 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.025987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.027024 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.027030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.029005 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.029021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.030613 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.030620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.030833 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.030858 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:20:40.030965 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 02:20:40.031127 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":940,"threads_started":1}
I20260926 02:20:40.032255 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c3516db0f543d888137ae9f41b7848" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" is_pre_election: true
I20260926 02:20:40.032567 21367 raft_consensus.cc:2454] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 0.
I20260926 02:20:40.033171 21013 leader_election.cc:302] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, cdfb3f232abc4c03af03ec3e6dfc6c02; no voters:
I20260926 02:20:40.033638 21443 raft_consensus.cc:2781] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:40.033766 21443 raft_consensus.cc:482] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:40.033896 21443 raft_consensus.cc:3037] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.037402 21443 raft_consensus.cc:504] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.038148 21443 leader_election.cc:288] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 election: Requested vote from peers e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017), cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615)
I20260926 02:20:40.039242 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.024282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:60587 (local address 127.12.244.194:41017)
0926 02:20:40.024531 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.024537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.031943 (+ 7406us) server_negotiation.cc:407] Connection header received
0926 02:20:40.032207 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.032212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.032303 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.032390 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:20:40.033033 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.034179 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.036635 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.037916 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.038081 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.038173 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.038732 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.038900 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:20:40.039558 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c3516db0f543d888137ae9f41b7848" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02"
I20260926 02:20:40.039801 21367 raft_consensus.cc:3037] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.039953 21479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.024684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:41017 (local address 127.12.244.193:60587)
0926 02:20:40.031855 (+ 7171us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:40.031876 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:40.032041 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:40.032645 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:40.032651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:40.032681 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:20:40.032878 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.032890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.034361 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.034369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.036438 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.036453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.038319 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.038325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.038524 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.038560 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:20:40.039490 (+ 930us) client_negotiation.cc:240] Negotiation successful
0926 02:20:40.039669 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7032,"threads_started":1}
I20260926 02:20:40.041945 21221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c3516db0f543d888137ae9f41b7848" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2" is_pre_election: true
I20260926 02:20:40.041990 21220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c3516db0f543d888137ae9f41b7848" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2"
I20260926 02:20:40.042271 21221 raft_consensus.cc:2454] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 0.
I20260926 02:20:40.043260 21367 raft_consensus.cc:2454] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 1.
I20260926 02:20:40.043781 21013 leader_election.cc:302] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, cdfb3f232abc4c03af03ec3e6dfc6c02; no voters: e8da33688e034d77b6cca969f18712d2
I20260926 02:20:40.044164 21443 raft_consensus.cc:2781] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:40.044759 21443 raft_consensus.cc:686] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [term 1 LEADER]: Becoming Leader. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Running, Role: LEADER
I20260926 02:20:40.045174 21443 consensus_queue.cc:237] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.048360 21481 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: TxnStatusTablet state changed. Reason: New leader b2e6b851830a4d54afd303ef7702966a. Latest consensus state: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.048702 21481 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:20:40.049427 21483 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:20:40.049662 21483 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:20:40.049341 20922 catalog_manager.cc:5495] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a reported cstate change: term changed from 0 to 1, leader changed from <none> to b2e6b851830a4d54afd303ef7702966a (127.12.244.193). New cstate: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } health_report { overall_health: UNKNOWN } } }
I20260926 02:20:40.080212 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.073848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.12.244.193:34521)
0926 02:20:40.074517 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.074522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.074538 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:20:40.074676 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.074681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.074791 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.074940 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:40.075357 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.076317 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.077855 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.078293 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.078451 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.078921 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.079547 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.079722 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.079798 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.079884 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547}
I20260926 02:20:40.080235 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.073784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:34521 (local address 127.0.0.1:41030)
0926 02:20:40.074102 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:40.074123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:40.074253 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:40.074958 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:40.074962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:40.074975 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:20:40.075209 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.075220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.076444 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.076449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.077704 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.077717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.078461 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.078467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.078650 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.078718 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.079667 (+ 949us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.079691 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:20:40.079779 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 02:20:40.079936 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 02:20:40.084489 21367 raft_consensus.cc:1261] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Refusing update from remote peer b2e6b851830a4d54afd303ef7702966a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.085278 21220 raft_consensus.cc:3037] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.086292 21481 consensus_queue.cc:1035] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.089974 21220 raft_consensus.cc:1261] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Refusing update from remote peer b2e6b851830a4d54afd303ef7702966a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.090785 21481 consensus_queue.cc:1035] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.100330 21447 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: TxnStatusTablet state changed. Reason: New leader b2e6b851830a4d54afd303ef7702966a. Latest consensus state: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.100693 21447 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:20:40.101802 21442 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: TxnStatusTablet state changed. Reason: New leader b2e6b851830a4d54afd303ef7702966a. Latest consensus state: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.102111 21442 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:20:40.102761 21481 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.103004 21481 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:20:40.103192 21452 mvcc.cc:205] Tried to move back new op lower bound from 7333434327378628608 to 7333434327229267968. Current Snapshot: MvccSnapshot[applied={T|T < 7333434327378628608 or (T in {7333434327378628608})}]
I20260926 02:20:40.112491 21447 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.112814 21447 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P e8da33688e034d77b6cca969f18712d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:20:40.121078 21442 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.121445 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.111247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.12.244.195:44615)
0926 02:20:40.111435 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.111444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.111467 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:20:40.111598 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.111606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.111817 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.112000 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:40.113736 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.115224 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.117791 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.118472 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.118619 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.119403 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.120206 (+ 803us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.120358 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.120884 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.120976 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:20:40.121487 21442 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P cdfb3f232abc4c03af03ec3e6dfc6c02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:20:40.125180 21489 tablet_replica.cc:439] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } }
I20260926 02:20:40.125566 21489 tablet_replica.cc:442] T f9c3516db0f543d888137ae9f41b7848 P b2e6b851830a4d54afd303ef7702966a: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:20:40.142593 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.134638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:38315 (local address 127.12.244.193:34521)
0926 02:20:40.134879 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.134887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.135735 (+ 848us) server_negotiation.cc:407] Connection header received
0926 02:20:40.135877 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.135883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.135975 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.136081 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:40.136684 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.137861 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.140025 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.140840 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.140999 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.141721 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.141874 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.141973 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.142156 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.142243 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:20:40.142684 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.134647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:34521 (local address 127.12.244.195:38315)
0926 02:20:40.135526 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:40.135552 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:40.135741 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:40.136127 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:40.136132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:40.136145 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:20:40.136494 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.136507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.138030 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.138036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.139853 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.139871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.141170 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.141177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.141394 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.141476 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.142026 (+ 550us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.142038 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:20:40.142124 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 02:20:40.142281 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":539,"threads_started":1}
I20260926 02:20:40.155330 21220 raft_consensus.cc:1261] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Refusing update from remote peer cdfb3f232abc4c03af03ec3e6dfc6c02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.155368 21080 raft_consensus.cc:1261] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Refusing update from remote peer cdfb3f232abc4c03af03ec3e6dfc6c02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.156235 21442 consensus_queue.cc:1035] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.156883 21442 consensus_queue.cc:1035] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.223070 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.207250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:42131 (local address 127.12.244.195:44615)
0926 02:20:40.207468 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.207475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.215894 (+ 8419us) server_negotiation.cc:407] Connection header received
0926 02:20:40.216220 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.216227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.216323 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.216388 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:40.217220 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.218363 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.220698 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.221484 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.221638 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.222224 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.222368 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.222434 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.222658 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.222751 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:20:40.223999 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.206664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44615 (local address 127.12.244.193:42131)
0926 02:20:40.215767 (+ 9103us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:40.215797 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:40.216035 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:40.216578 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:40.216583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:40.216595 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:20:40.216974 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.216986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.218551 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.218557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.220505 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.220523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.221708 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.221716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.221901 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.221986 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.222515 (+ 529us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.222525 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:20:40.223372 (+ 847us) client_negotiation.cc:240] Negotiation successful
0926 02:20:40.223587 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8833,"thread_start_us":8642,"threads_started":1}
I20260926 02:20:40.306690 20919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:40.306990 20919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:40.310909 20919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Sending DeleteTablet for 3 replicas of tablet ebbb474a921e471f95288126eceea84e
I20260926 02:20:40.312387 21059 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbb474a921e471f95288126eceea84e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41022
I20260926 02:20:40.312721 21201 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbb474a921e471f95288126eceea84e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:58266
I20260926 02:20:40.313083 21460 tablet_replica.cc:330] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: stopping tablet replica
I20260926 02:20:40.313285 21460 raft_consensus.cc:2229] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:40.313489 21460 raft_consensus.cc:2258] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.314750 21342 tablet_service.cc:1514] Processing DeleteTablet for tablet ebbb474a921e471f95288126eceea84e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:43190
I20260926 02:20:40.315050 21459 tablet_replica.cc:330] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: stopping tablet replica
I20260926 02:20:40.315274 21459 raft_consensus.cc:2229] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:40.315786 21459 raft_consensus.cc:2258] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.318056 21460 ts_tablet_manager.cc:1905] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.324113 21459 ts_tablet_manager.cc:1905] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.339586 21459 ts_tablet_manager.cc:1918] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:20:40.339834 21459 log.cc:1196] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/ebbb474a921e471f95288126eceea84e
I20260926 02:20:40.340387 21459 ts_tablet_manager.cc:1939] T ebbb474a921e471f95288126eceea84e P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting consensus metadata
I20260926 02:20:40.341826 20909 catalog_manager.cc:4841] TS cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615): tablet ebbb474a921e471f95288126eceea84e (table test-table [id=4a8d5379dea74e379cd8807092a50b03]) successfully deleted
I20260926 02:20:40.344130 21460 ts_tablet_manager.cc:1918] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:20:40.344312 21460 log.cc:1196] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/ebbb474a921e471f95288126eceea84e
I20260926 02:20:40.345184 21460 ts_tablet_manager.cc:1939] T ebbb474a921e471f95288126eceea84e P b2e6b851830a4d54afd303ef7702966a: Deleting consensus metadata
I20260926 02:20:40.346031 21499 tablet_replica.cc:330] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: stopping tablet replica
I20260926 02:20:40.346251 21499 raft_consensus.cc:2229] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:40.346491 21499 raft_consensus.cc:2258] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.346738 20908 catalog_manager.cc:4841] TS b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521): tablet ebbb474a921e471f95288126eceea84e (table test-table [id=4a8d5379dea74e379cd8807092a50b03]) successfully deleted
I20260926 02:20:40.347694 21499 ts_tablet_manager.cc:1905] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.355208 21499 ts_tablet_manager.cc:1918] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:20:40.355381 21499 log.cc:1196] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/ebbb474a921e471f95288126eceea84e
I20260926 02:20:40.355858 21499 ts_tablet_manager.cc:1939] T ebbb474a921e471f95288126eceea84e P e8da33688e034d77b6cca969f18712d2: Deleting consensus metadata
I20260926 02:20:40.357087 20907 catalog_manager.cc:4841] TS e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017): tablet ebbb474a921e471f95288126eceea84e (table test-table [id=4a8d5379dea74e379cd8807092a50b03]) successfully deleted
Sep 26 02:20:40 dist-test-slave-qfmw krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:20:40 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:40 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:20:40.453315 21300 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:40.439594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.12.244.254:43003)
0926 02:20:40.439763 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.439793 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.440315 (+ 522us) server_negotiation.cc:407] Connection header received
0926 02:20:40.440503 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.440509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.440596 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.440716 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:40.442475 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.443731 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.445739 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.446470 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.446610 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.452998 (+ 6388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33036: BlockingRecv error: recv got EOF from 127.0.0.1:33036 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 02:20:40 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM
I20260926 02:20:40.484678 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.465024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.12.244.254:43003)
0926 02:20:40.465237 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.465243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.465869 (+ 626us) server_negotiation.cc:407] Connection header received
0926 02:20:40.466080 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.466086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.466192 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.466306 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:40.468076 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.469200 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.471124 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.471790 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.471940 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.477429 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:40.477437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:40.477442 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:40.477507 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:40.481347 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:40.482310 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:40.482319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:40.482323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:40.482490 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:40.482959 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:40.482964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:40.482968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:40.483407 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:40.483686 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.484033 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.484298 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:20:40.489183 20919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:20:40.489988 20919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:40.498775 21059 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7a94d6fefe47ee9bfea21ddaee4408 (DEFAULT_TABLE table=test-table [id=137cc3fe29c24405ad28d2eeb46b6fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:40.499352 21059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7a94d6fefe47ee9bfea21ddaee4408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:40.499447 21342 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7a94d6fefe47ee9bfea21ddaee4408 (DEFAULT_TABLE table=test-table [id=137cc3fe29c24405ad28d2eeb46b6fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:40.499881 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7a94d6fefe47ee9bfea21ddaee4408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:40.503566 21201 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7a94d6fefe47ee9bfea21ddaee4408 (DEFAULT_TABLE table=test-table [id=137cc3fe29c24405ad28d2eeb46b6fc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:40.504026 21201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7a94d6fefe47ee9bfea21ddaee4408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:40.509814 21437 tablet_bootstrap.cc:492] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: Bootstrap starting.
I20260926 02:20:40.511504 21437 tablet_bootstrap.cc:654] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:40.513562 21437 tablet_bootstrap.cc:492] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: No bootstrap required, opened a new log
I20260926 02:20:40.513746 21437 ts_tablet_manager.cc:1397] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:40.514453 21437 raft_consensus.cc:348] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.514683 21437 raft_consensus.cc:374] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.514771 21437 raft_consensus.cc:729] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.514997 21438 tablet_bootstrap.cc:492] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: Bootstrap starting.
I20260926 02:20:40.515030 21437 consensus_queue.cc:260] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.516196 21437 ts_tablet_manager.cc:1428] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:20:40.516722 21438 tablet_bootstrap.cc:654] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:40.519599 21438 tablet_bootstrap.cc:492] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: No bootstrap required, opened a new log
I20260926 02:20:40.519755 21438 ts_tablet_manager.cc:1397] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:20:40.520527 21438 raft_consensus.cc:348] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.520789 21438 raft_consensus.cc:374] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.520941 21438 raft_consensus.cc:729] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8da33688e034d77b6cca969f18712d2, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.522481 21438 consensus_queue.cc:260] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.525758 21438 ts_tablet_manager.cc:1428] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 02:20:40.530138 21513 tablet_bootstrap.cc:492] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Bootstrap starting.
I20260926 02:20:40.532238 21513 tablet_bootstrap.cc:654] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:40.540031 21513 tablet_bootstrap.cc:492] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: No bootstrap required, opened a new log
I20260926 02:20:40.540491 21513 ts_tablet_manager.cc:1397] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260926 02:20:40.541430 21513 raft_consensus.cc:348] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.541718 21513 raft_consensus.cc:374] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.541836 21513 raft_consensus.cc:729] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfb3f232abc4c03af03ec3e6dfc6c02, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.542138 21513 consensus_queue.cc:260] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.542799 21513 ts_tablet_manager.cc:1428] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:20:40.607585 21489 raft_consensus.cc:482] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:40.607820 21489 raft_consensus.cc:504] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.608515 21489 leader_election.cc:288] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017), cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615)
I20260926 02:20:40.609104 21220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7a94d6fefe47ee9bfea21ddaee4408" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2" is_pre_election: true
I20260926 02:20:40.609226 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7a94d6fefe47ee9bfea21ddaee4408" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" is_pre_election: true
I20260926 02:20:40.609360 21220 raft_consensus.cc:2454] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 0.
I20260926 02:20:40.609500 21367 raft_consensus.cc:2454] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 0.
I20260926 02:20:40.609972 21013 leader_election.cc:302] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, cdfb3f232abc4c03af03ec3e6dfc6c02; no voters:
I20260926 02:20:40.610245 21489 raft_consensus.cc:2781] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:40.610350 21489 raft_consensus.cc:482] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:40.610422 21489 raft_consensus.cc:3037] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.613449 21489 raft_consensus.cc:504] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.614647 21220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7a94d6fefe47ee9bfea21ddaee4408" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2"
I20260926 02:20:40.614887 21220 raft_consensus.cc:3037] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.617861 21220 raft_consensus.cc:2454] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 1.
I20260926 02:20:40.618412 21011 leader_election.cc:302] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, e8da33688e034d77b6cca969f18712d2; no voters:
I20260926 02:20:40.618772 21481 raft_consensus.cc:2781] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:40.618911 21481 raft_consensus.cc:686] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 1 LEADER]: Becoming Leader. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Running, Role: LEADER
I20260926 02:20:40.619238 21481 consensus_queue.cc:237] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } }
I20260926 02:20:40.621997 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7a94d6fefe47ee9bfea21ddaee4408" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02"
I20260926 02:20:40.622225 21367 raft_consensus.cc:3037] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.625604 21367 raft_consensus.cc:2454] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 1.
I20260926 02:20:40.627887 20919 catalog_manager.cc:5495] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a reported cstate change: term changed from 0 to 1, leader changed from <none> to b2e6b851830a4d54afd303ef7702966a (127.12.244.193). New cstate: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } health_report { overall_health: UNKNOWN } } }
I20260926 02:20:40.630503 21489 leader_election.cc:288] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 election: Requested vote from peers e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017), cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615)
I20260926 02:20:40.672444 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.666263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.12.244.193:34521)
0926 02:20:40.666772 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.666778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.666808 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:20:40.666883 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.666890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.666980 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.667124 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:40.667538 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.668580 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.670138 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.670741 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.670920 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.671529 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.671686 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.671793 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.671989 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.672079 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
I20260926 02:20:40.676527 21220 raft_consensus.cc:1261] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Refusing update from remote peer b2e6b851830a4d54afd303ef7702966a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.676952 21367 raft_consensus.cc:1261] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Refusing update from remote peer b2e6b851830a4d54afd303ef7702966a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.677743 21489 consensus_queue.cc:1035] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.678074 21481 consensus_queue.cc:1035] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.684857 21450 mvcc.cc:205] Tried to move back new op lower bound from 7333434329803014144 to 7333434329579048960. Current Snapshot: MvccSnapshot[applied={T|T < 7333434329803014144}]
I20260926 02:20:40.693127 20919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:40.693427 20919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:40.697813 20919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Sending DeleteTablet for 3 replicas of tablet 0d7a94d6fefe47ee9bfea21ddaee4408
I20260926 02:20:40.698719 21059 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d7a94d6fefe47ee9bfea21ddaee4408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41022
I20260926 02:20:40.699069 21460 tablet_replica.cc:330] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: stopping tablet replica
I20260926 02:20:40.699325 21460 raft_consensus.cc:2229] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:40.699882 21460 raft_consensus.cc:2258] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.701189 21460 ts_tablet_manager.cc:1905] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.707531 21201 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d7a94d6fefe47ee9bfea21ddaee4408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:58266
I20260926 02:20:40.707897 21499 tablet_replica.cc:330] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: stopping tablet replica
I20260926 02:20:40.708122 21499 raft_consensus.cc:2229] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:40.708391 21499 raft_consensus.cc:2258] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.709721 21499 ts_tablet_manager.cc:1905] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.714392 21342 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d7a94d6fefe47ee9bfea21ddaee4408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:43190
I20260926 02:20:40.714735 21459 tablet_replica.cc:330] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: stopping tablet replica
I20260926 02:20:40.716477 21460 ts_tablet_manager.cc:1918] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:40.716722 21460 log.cc:1196] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/0d7a94d6fefe47ee9bfea21ddaee4408
I20260926 02:20:40.717288 21460 ts_tablet_manager.cc:1939] T 0d7a94d6fefe47ee9bfea21ddaee4408 P b2e6b851830a4d54afd303ef7702966a: Deleting consensus metadata
I20260926 02:20:40.719837 20919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:20:40.720077 20908 catalog_manager.cc:4841] TS b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521): tablet 0d7a94d6fefe47ee9bfea21ddaee4408 (table test-table [id=137cc3fe29c24405ad28d2eeb46b6fc3]) successfully deleted
W20260926 02:20:40.720589 20919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:20:40.723563 21459 raft_consensus.cc:2229] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:40.724161 21459 pending_rounds.cc:62] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Trying to abort 2 pending ops.
I20260926 02:20:40.724299 21459 pending_rounds.cc:68] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333434329803014144 op_type: WRITE_OP write_request { tablet_id: "0d7a94d6fefe47ee9bfea21ddaee4408" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\244\332\334\325\006\"8\n\ntest-admin\022*\n 137cc3fe29c24405ad28d2eeb46b6fc3\020\001\030\001 \001(\001" signature: "<redacted>""]z\0048n\002\206@!{{Z\302u\226\200w~\3169\\l\221\272\377\237\312\'9|\021\2466\017\330\013\247F\200\335L\341\'\006\374\007FO\364b\004\256\214\332*\273\310H\360=\202k\206~" signing_key_seq_num: 0 } } request_id { client_id: "0354d8e8e70a42459af07982e0a383b2" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 02:20:40.725020 21459 pending_rounds.cc:68] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333434329579048960 op_type: NO_OP noop_request { }
I20260926 02:20:40.725193 21459 raft_consensus.cc:2866] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 02:20:40.725355 21459 raft_consensus.cc:2258] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.719024 21499 ts_tablet_manager.cc:1918] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 02:20:40.726786 21520 consensus_queue.cc:1159] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260926 02:20:40.727072 21499 log.cc:1196] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/0d7a94d6fefe47ee9bfea21ddaee4408
I20260926 02:20:40.728170 21499 ts_tablet_manager.cc:1939] T 0d7a94d6fefe47ee9bfea21ddaee4408 P e8da33688e034d77b6cca969f18712d2: Deleting consensus metadata
I20260926 02:20:40.728394 21459 ts_tablet_manager.cc:1905] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.729394 20907 catalog_manager.cc:4841] TS e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017): tablet 0d7a94d6fefe47ee9bfea21ddaee4408 (table test-table [id=137cc3fe29c24405ad28d2eeb46b6fc3]) successfully deleted
I20260926 02:20:40.736027 21459 ts_tablet_manager.cc:1918] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:20:40.736263 21342 tablet_service.cc:1467] Processing CreateTablet for tablet 5424c05b89e641b28f108f20e4547ea2 (DEFAULT_TABLE table=test-table [id=32e8b21f7c764cdeba1aee045ecf4a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:40.736799 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5424c05b89e641b28f108f20e4547ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:40.736681 21059 tablet_service.cc:1467] Processing CreateTablet for tablet 5424c05b89e641b28f108f20e4547ea2 (DEFAULT_TABLE table=test-table [id=32e8b21f7c764cdeba1aee045ecf4a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:40.737181 21059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5424c05b89e641b28f108f20e4547ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:40.744086 21201 tablet_service.cc:1467] Processing CreateTablet for tablet 5424c05b89e641b28f108f20e4547ea2 (DEFAULT_TABLE table=test-table [id=32e8b21f7c764cdeba1aee045ecf4a7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:20:40.744350 21513 tablet_bootstrap.cc:492] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: Bootstrap starting.
I20260926 02:20:40.744583 21201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5424c05b89e641b28f108f20e4547ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:40.746158 21513 tablet_bootstrap.cc:654] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:40.736299 21459 log.cc:1196] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/0d7a94d6fefe47ee9bfea21ddaee4408
I20260926 02:20:40.751601 21459 ts_tablet_manager.cc:1939] T 0d7a94d6fefe47ee9bfea21ddaee4408 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting consensus metadata
I20260926 02:20:40.752794 21437 tablet_bootstrap.cc:492] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: Bootstrap starting.
I20260926 02:20:40.753839 21438 tablet_bootstrap.cc:492] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: Bootstrap starting.
I20260926 02:20:40.754739 21437 tablet_bootstrap.cc:654] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:40.755581 21438 tablet_bootstrap.cc:654] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:40.764533 20909 catalog_manager.cc:4841] TS cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615): tablet 0d7a94d6fefe47ee9bfea21ddaee4408 (table test-table [id=137cc3fe29c24405ad28d2eeb46b6fc3]) successfully deleted
I20260926 02:20:40.764631 21438 tablet_bootstrap.cc:492] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: No bootstrap required, opened a new log
I20260926 02:20:40.764813 21438 ts_tablet_manager.cc:1397] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260926 02:20:40.765704 21438 raft_consensus.cc:348] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.765882 21513 tablet_bootstrap.cc:492] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: No bootstrap required, opened a new log
I20260926 02:20:40.765942 21438 raft_consensus.cc:374] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.766016 21438 raft_consensus.cc:729] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8da33688e034d77b6cca969f18712d2, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.766039 21513 ts_tablet_manager.cc:1397] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20260926 02:20:40.766299 21438 consensus_queue.cc:260] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.766789 21438 ts_tablet_manager.cc:1428] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 02:20:40.766796 21513 raft_consensus.cc:348] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.767053 21513 raft_consensus.cc:374] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.769552 21513 raft_consensus.cc:729] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdfb3f232abc4c03af03ec3e6dfc6c02, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.770104 21513 consensus_queue.cc:260] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.772557 21513 ts_tablet_manager.cc:1428] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260926 02:20:40.776075 21437 tablet_bootstrap.cc:492] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: No bootstrap required, opened a new log
I20260926 02:20:40.776280 21437 ts_tablet_manager.cc:1397] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.004s
I20260926 02:20:40.777149 21437 raft_consensus.cc:348] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.777432 21437 raft_consensus.cc:374] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:40.777516 21437 raft_consensus.cc:729] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Initialized, Role: FOLLOWER
I20260926 02:20:40.777793 21437 consensus_queue.cc:260] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.778956 21437 ts_tablet_manager.cc:1428] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 02:20:40.802403 21489 raft_consensus.cc:482] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:20:40.802657 21489 raft_consensus.cc:504] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.803465 21489 leader_election.cc:288] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615), e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017)
I20260926 02:20:40.803987 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5424c05b89e641b28f108f20e4547ea2" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" is_pre_election: true
I20260926 02:20:40.804206 21220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5424c05b89e641b28f108f20e4547ea2" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2" is_pre_election: true
I20260926 02:20:40.804250 21367 raft_consensus.cc:2454] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 0.
I20260926 02:20:40.804454 21220 raft_consensus.cc:2454] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 0.
I20260926 02:20:40.804934 21011 leader_election.cc:302] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, e8da33688e034d77b6cca969f18712d2; no voters:
I20260926 02:20:40.805282 21489 raft_consensus.cc:2781] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:20:40.805395 21489 raft_consensus.cc:482] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:20:40.805521 21489 raft_consensus.cc:3037] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.809724 21489 raft_consensus.cc:504] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.810957 21367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5424c05b89e641b28f108f20e4547ea2" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02"
I20260926 02:20:40.811010 21220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5424c05b89e641b28f108f20e4547ea2" candidate_uuid: "b2e6b851830a4d54afd303ef7702966a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8da33688e034d77b6cca969f18712d2"
I20260926 02:20:40.811210 21367 raft_consensus.cc:3037] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.811252 21220 raft_consensus.cc:3037] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:40.814451 21367 raft_consensus.cc:2454] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 1.
I20260926 02:20:40.814464 21220 raft_consensus.cc:2454] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2e6b851830a4d54afd303ef7702966a in term 1.
I20260926 02:20:40.814985 21011 leader_election.cc:302] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2e6b851830a4d54afd303ef7702966a, e8da33688e034d77b6cca969f18712d2; no voters:
I20260926 02:20:40.815295 21443 raft_consensus.cc:2781] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:40.815477 21443 raft_consensus.cc:686] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 1 LEADER]: Becoming Leader. State: Replica: b2e6b851830a4d54afd303ef7702966a, State: Running, Role: LEADER
I20260926 02:20:40.815717 21443 consensus_queue.cc:237] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } }
I20260926 02:20:40.816371 21489 leader_election.cc:288] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [CANDIDATE]: Term 1 election: Requested vote from peers cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615), e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017)
I20260926 02:20:40.824894 20919 catalog_manager.cc:5495] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a reported cstate change: term changed from 0 to 1, leader changed from <none> to b2e6b851830a4d54afd303ef7702966a (127.12.244.193). New cstate: current_term: 1 leader_uuid: "b2e6b851830a4d54afd303ef7702966a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e6b851830a4d54afd303ef7702966a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 34521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 } health_report { overall_health: UNKNOWN } } }
I20260926 02:20:40.875149 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.868546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:49091 (local address 127.12.244.193:34521)
0926 02:20:40.868755 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.868761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.869370 (+ 609us) server_negotiation.cc:407] Connection header received
0926 02:20:40.869580 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.869585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.869655 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.869755 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:20:40.870254 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.871067 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.872896 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.873439 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.873605 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.874107 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.874232 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.874332 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:40.874431 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 02:20:40.874635 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":51}
I20260926 02:20:40.875169 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.868565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:34521 (local address 127.12.244.193:49091)
0926 02:20:40.869257 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:40.869275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:40.869446 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:40.869765 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:40.869769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:40.869779 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:20:40.870091 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.870100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.871256 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.871261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.872637 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:40.872651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.873661 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.873673 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:40.873837 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.873918 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:20:40.874330 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:20:40.874340 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:20:40.874418 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 02:20:40.874656 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":2,"thread_start_us":359,"threads_started":1}
I20260926 02:20:40.886929 21367 raft_consensus.cc:1261] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Refusing update from remote peer b2e6b851830a4d54afd303ef7702966a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.887087 21220 raft_consensus.cc:1261] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Refusing update from remote peer b2e6b851830a4d54afd303ef7702966a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:20:40.887786 21443 consensus_queue.cc:1035] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8da33688e034d77b6cca969f18712d2" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.888087 21489 consensus_queue.cc:1035] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdfb3f232abc4c03af03ec3e6dfc6c02" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:20:40.957129 20922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:40.957446 20922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:20:40.960915 20922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9561fffef7a946e696e8d3b05ce62291: Sending DeleteTablet for 3 replicas of tablet 5424c05b89e641b28f108f20e4547ea2
I20260926 02:20:40.961781 21342 tablet_service.cc:1514] Processing DeleteTablet for tablet 5424c05b89e641b28f108f20e4547ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:43190
I20260926 02:20:40.961809 21059 tablet_service.cc:1514] Processing DeleteTablet for tablet 5424c05b89e641b28f108f20e4547ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41022
I20260926 02:20:40.962095 21460 tablet_replica.cc:330] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: stopping tablet replica
I20260926 02:20:40.962090 21459 tablet_replica.cc:330] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: stopping tablet replica
I20260926 02:20:40.962064 21201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5424c05b89e641b28f108f20e4547ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:20:40 UTC) from {username='oryx', principal='oryx/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:58266
I20260926 02:20:40.962283 21460 raft_consensus.cc:2229] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:20:40.962280 21459 raft_consensus.cc:2229] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:40.962370 21499 tablet_replica.cc:330] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: stopping tablet replica
I20260926 02:20:40.962488 21459 raft_consensus.cc:2258] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.962544 21499 raft_consensus.cc:2229] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:20:40.962622 21460 raft_consensus.cc:2258] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.962766 21499 raft_consensus.cc:2258] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:20:40.964058 21499 ts_tablet_manager.cc:1905] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.964206 21459 ts_tablet_manager.cc:1905] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.967962 21460 ts_tablet_manager.cc:1905] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:20:40.969853 21499 ts_tablet_manager.cc:1918] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:20:40.969995 21499 log.cc:1196] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/5424c05b89e641b28f108f20e4547ea2
I20260926 02:20:40.970449 21499 ts_tablet_manager.cc:1939] T 5424c05b89e641b28f108f20e4547ea2 P e8da33688e034d77b6cca969f18712d2: Deleting consensus metadata
I20260926 02:20:40.971594 21459 ts_tablet_manager.cc:1918] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:20:40.971674 20907 catalog_manager.cc:4841] TS e8da33688e034d77b6cca969f18712d2 (127.12.244.194:41017): tablet 5424c05b89e641b28f108f20e4547ea2 (table test-table [id=32e8b21f7c764cdeba1aee045ecf4a7d]) successfully deleted
I20260926 02:20:40.971767 21459 log.cc:1196] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/5424c05b89e641b28f108f20e4547ea2
I20260926 02:20:40.972179 21459 ts_tablet_manager.cc:1939] T 5424c05b89e641b28f108f20e4547ea2 P cdfb3f232abc4c03af03ec3e6dfc6c02: Deleting consensus metadata
I20260926 02:20:40.973330 20909 catalog_manager.cc:4841] TS cdfb3f232abc4c03af03ec3e6dfc6c02 (127.12.244.195:44615): tablet 5424c05b89e641b28f108f20e4547ea2 (table test-table [id=32e8b21f7c764cdeba1aee045ecf4a7d]) successfully deleted
I20260926 02:20:40.974727 21460 ts_tablet_manager.cc:1918] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:20:40.974862 21460 log.cc:1196] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/5424c05b89e641b28f108f20e4547ea2
I20260926 02:20:40.975426 21460 ts_tablet_manager.cc:1939] T 5424c05b89e641b28f108f20e4547ea2 P b2e6b851830a4d54afd303ef7702966a: Deleting consensus metadata
I20260926 02:20:40.976606 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 20994
I20260926 02:20:40.976668 20908 catalog_manager.cc:4841] TS b2e6b851830a4d54afd303ef7702966a (127.12.244.193:34521): tablet 5424c05b89e641b28f108f20e4547ea2 (table test-table [id=32e8b21f7c764cdeba1aee045ecf4a7d]) successfully deleted
Sep 26 02:20:40 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.193@KRBTEST.COM
I20260926 02:20:41.003368 21518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:40.977744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41040 (local address 127.12.244.193:34521)
0926 02:20:40.977936 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:40.977942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:40.985842 (+ 7900us) server_negotiation.cc:407] Connection header received
0926 02:20:40.985994 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:40.986000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:40.986098 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:40.986229 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:40.987602 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.988849 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.990324 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:40.991016 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:40.991255 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:40.995548 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:40.995571 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:40.995591 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:40.995638 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:40.999772 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:41.000689 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:41.000701 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:41.000705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:41.000938 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:41.001473 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:41.001478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:41.001481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:41.001936 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:41.002298 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:41.002684 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 02:20:41.002934 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:20:41.004149 21126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:41.152673 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 20994
I20260926 02:20:41.177724 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 21135
Sep 26 02:20:41 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.194@KRBTEST.COM
I20260926 02:20:41.194489 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:41.178549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.12.244.194:41017)
0926 02:20:41.179188 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:41.179197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:41.179224 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:20:41.179287 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:41.179291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:41.179355 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:41.179461 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:41.180791 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:41.181555 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:41.182873 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:41.183488 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:41.183627 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:41.188113 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:41.188130 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:41.188147 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:41.188188 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:41.191445 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:41.192190 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:41.192198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:41.192200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:41.192345 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:41.192745 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:41.192749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:41.192751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:41.193091 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:41.193300 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:41.193858 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 02:20:41.194152 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":280,"threads_started":1}
I20260926 02:20:41.195206 21267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:41.318219 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 21135
I20260926 02:20:41.341046 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 21276
Sep 26 02:20:41 dist-test-slave-qfmw krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.244.195@KRBTEST.COM
I20260926 02:20:41.358013 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:41.341790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.12.244.195:44615)
0926 02:20:41.342305 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:41.342310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:41.342325 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:20:41.342375 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:41.342379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:41.342442 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:41.342563 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:41.344010 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:41.344719 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:41.346044 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:41.346610 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:41.346762 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:41.351882 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:41.351899 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:41.351917 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:41.351964 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:41.354910 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:41.355752 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:41.355760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:41.355764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:41.355903 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:41.356328 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:41.356332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:41.356334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:41.356677 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:41.356977 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:41.357354 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 02:20:41.357609 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":255,"threads_started":1}
I20260926 02:20:41.358780 21407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:41.483489 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 21276
I20260926 02:20:41.501511 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 20890
I20260926 02:20:41.502631 20962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:41.618986 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 20890
2026-09-26T02:20:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5818 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:41 dist-test-slave-qfmw krb5kdc[21542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:41 dist-test-slave-qfmw krb5kdc[21542](info): set up 2 sockets
Sep 26 02:20:41 dist-test-slave-qfmw krb5kdc[21542](info): commencing operation
krb5kdc: starting...
W20260926 02:20:42.742787 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:42 dist-test-slave-qfmw krb5kdc[21542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:20:42.987944 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:42759
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:43.366786 21555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:43.367250 21555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:43.367429 21555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:43.375038 21555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:43.375279 21555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:43.375345 21555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:43.375412 21555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:43.375458 21555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:43.385775 21555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:42759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:43.387136 21555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:43.388520 21555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:43.401803 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:43.401924 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:43.401875 21562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:43.405227 21555 server_base.cc:1034] running on GCE node
I20260926 02:20:43.406296 21555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:43.408102 21555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:43.409400 21555 hybrid_clock.cc:648] HybridClock initialized: now 1790389243409350 us; error 74 us; skew 500 ppm
I20260926 02:20:43.409876 21555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T02:20:43Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1860 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:20:43 dist-test-slave-qfmw krb5kdc[21570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:20:43 dist-test-slave-qfmw krb5kdc[21570](info): set up 2 sockets
Sep 26 02:20:43 dist-test-slave-qfmw krb5kdc[21570](info): commencing operation
krb5kdc: starting...
W20260926 02:20:44.638722 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:20:44 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:20:44.879434 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:41137
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:45.230851 21583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:45.231261 21583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:45.231446 21583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:45.239025 21583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:45.239264 21583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:45.239383 21583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:45.239475 21583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:45.239543 21583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:45.250018 21583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:41137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:45.251600 21583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:45.253049 21583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:45.265794 21591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:45.265877 21589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:45.265821 21588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:45.266532 21583 server_base.cc:1034] running on GCE node
I20260926 02:20:45.267601 21583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:45.269253 21583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:45.270498 21583 hybrid_clock.cc:648] HybridClock initialized: now 1790389245270440 us; error 49 us; skew 500 ppm
I20260926 02:20:45.270901 21583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:45 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389245, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:45.282238 21583 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:20:45.284080 21583 webserver.cc:460] Webserver started at http://127.12.244.254:32957/ using document root <none> and password file <none>
I20260926 02:20:45.284926 21583 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:45.285091 21583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:45.285468 21583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:45.288869 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "2c7f2f2f8cdd42029915164b9834bb16"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "86c563959d06e1328c789b055efc5e18"
server_key_iv: "af33f5856e2e2a958b40ba6e813f689a"
server_key_version: "encryptionkey@0"
I20260926 02:20:45.289769 21583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "2c7f2f2f8cdd42029915164b9834bb16"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "86c563959d06e1328c789b055efc5e18"
server_key_iv: "af33f5856e2e2a958b40ba6e813f689a"
server_key_version: "encryptionkey@0"
I20260926 02:20:45.295698 21583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 02:20:45.299811 21598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:45.301503 21583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:20:45.301733 21583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "2c7f2f2f8cdd42029915164b9834bb16"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "86c563959d06e1328c789b055efc5e18"
server_key_iv: "af33f5856e2e2a958b40ba6e813f689a"
server_key_version: "encryptionkey@0"
I20260926 02:20:45.301965 21583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:45.323189 21583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:45.328073 21583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:45.328410 21583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:45.348333 21583 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:41137
I20260926 02:20:45.348362 21649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:41137 every 8 connection(s)
I20260926 02:20:45.349898 21583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:45.353464 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 21583
I20260926 02:20:45.353734 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:20:45.354087 21650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:20:45.354458 13267 external_mini_cluster.cc:1428] Setting key acef49bfb72ccb18a652b12f74d67432
I20260926 02:20:45.370509 21650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Bootstrap starting.
Sep 26 02:20:45 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:45.374269 21650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Neither blocks nor log segments found. Creating new log.
I20260926 02:20:45.375540 21650 log.cc:824] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:45.379500 21650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: No bootstrap required, opened a new log
I20260926 02:20:45.381623 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:45.356834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.12.244.254:41137)
0926 02:20:45.357847 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:45.357860 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:45.357913 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:20:45.359896 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:45.359924 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:45.360401 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:45.360803 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:45.362869 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:45.364487 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:45.366475 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:45.367465 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:45.367638 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:45.372492 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:45.372501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:45.372518 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:45.372566 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:45.376303 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:45.377360 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:45.377370 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:45.377380 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:45.377608 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:45.378043 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:45.378051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:45.378057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:45.378855 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:45.379167 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:45.379677 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 02:20:45.380184 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":474,"threads_started":1}
I20260926 02:20:45.389108 21650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:45.389503 21650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:20:45.389582 21650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7f2f2f8cdd42029915164b9834bb16, State: Initialized, Role: FOLLOWER
I20260926 02:20:45.390185 21650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:45.390396 21650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:45.390486 21650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:45.390614 21650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:20:45.392923 21650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:45.393428 21650 leader_election.cc:302] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c7f2f2f8cdd42029915164b9834bb16; no voters:
I20260926 02:20:45.394132 21650 leader_election.cc:288] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:20:45.394312 21655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:20:45.396517 21650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:45.398574 21655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 LEADER]: Becoming Leader. State: Replica: 2c7f2f2f8cdd42029915164b9834bb16, State: Running, Role: LEADER
I20260926 02:20:45.399343 21655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:45.402923 21657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c7f2f2f8cdd42029915164b9834bb16. Latest consensus state: current_term: 1 leader_uuid: "2c7f2f2f8cdd42029915164b9834bb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } } }
I20260926 02:20:45.403384 21657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:45.403483 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c7f2f2f8cdd42029915164b9834bb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } } }
I20260926 02:20:45.403802 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:45.408748 21669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:45.413590 21669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:45.422991 21669 catalog_manager.cc:1347] Generated new cluster ID: 5ccbf071c6504f37b826fa36f11cc549
I20260926 02:20:45.423161 21669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:45.433730 21669 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:20:45.435853 21669 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:45.454219 21669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Generated new TSK 0
I20260926 02:20:45.455546 21669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:20:45.527501 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:41137
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:45.889734 21676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:45.890120 21676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:45.890295 21676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:45.898113 21676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:45.898303 21676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:45.898491 21676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:45.909499 21676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:45.910984 21676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:45.912477 21676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:45.927155 21681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:45.927084 21682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:45.927181 21684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:45.928188 21676 server_base.cc:1034] running on GCE node
I20260926 02:20:45.928990 21676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:45.930159 21676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:45.931442 21676 hybrid_clock.cc:648] HybridClock initialized: now 1790389245931385 us; error 47 us; skew 500 ppm
I20260926 02:20:45.931926 21676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:45 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389245, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:45.935853 21676 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:20:45.937673 21676 webserver.cc:460] Webserver started at http://127.12.244.193:38905/ using document root <none> and password file <none>
I20260926 02:20:45.938511 21676 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:45.938661 21676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:45.939043 21676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:45.942441 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "56908119d2b5428193e2862a08bc13bc"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "0caa1a32bd8c41b92467e23fa0275c8d"
server_key_iv: "8ea616c54f6f08abb42e29041b3db00d"
server_key_version: "encryptionkey@0"
I20260926 02:20:45.943867 21676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "56908119d2b5428193e2862a08bc13bc"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "0caa1a32bd8c41b92467e23fa0275c8d"
server_key_iv: "8ea616c54f6f08abb42e29041b3db00d"
server_key_version: "encryptionkey@0"
I20260926 02:20:45.950135 21676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:20:45.954417 21691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:45.956486 21676 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:45.956758 21676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "56908119d2b5428193e2862a08bc13bc"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "0caa1a32bd8c41b92467e23fa0275c8d"
server_key_iv: "8ea616c54f6f08abb42e29041b3db00d"
server_key_version: "encryptionkey@0"
I20260926 02:20:45.957026 21676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:45.983028 21676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:45.988036 21676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:45.988309 21676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:45.989354 21676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:45.990783 21676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:45.990870 21676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:45.990962 21676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:45.991010 21676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:46.026388 21676 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45109
I20260926 02:20:46.026516 21805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45109 every 8 connection(s)
I20260926 02:20:46.027889 21676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:46.034665 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 21676
I20260926 02:20:46.034889 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:20:46.035602 13267 external_mini_cluster.cc:1428] Setting key 2680301897a66b930e4dc8158a0d76a7
Sep 26 02:20:46 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389245, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:46.051548 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:46.030693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:49981 (local address 127.12.244.254:41137)
0926 02:20:46.031237 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:46.031242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:46.032363 (+ 1121us) server_negotiation.cc:407] Connection header received
0926 02:20:46.034392 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:46.034398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:46.034502 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:46.034628 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:46.036735 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.037663 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.039460 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.039886 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.040031 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:46.043982 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:46.043988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:46.043992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:46.044029 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:46.047594 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:46.048571 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:46.048581 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:46.048587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:46.048758 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:46.049251 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:46.049259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:46.049265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:46.049664 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:46.049822 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:46.050703 (+ 881us) server_negotiation.cc:299] Negotiation successful
0926 02:20:46.050981 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":249,"threads_started":1}
I20260926 02:20:46.052366 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:46.031372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:49981)
0926 02:20:46.032165 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:46.032207 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:46.034078 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:46.034985 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:46.035000 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:46.035682 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:46.036546 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:46.036570 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.037838 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.037844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:46.039290 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:46.039304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.040153 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.040160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:46.040411 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:46.041172 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:46.041196 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:46.043651 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:46.047816 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:46.047826 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:46.047844 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:46.048396 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:46.048911 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:46.048914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:46.048916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:46.049120 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:46.049818 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:46.049825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:46.050264 (+ 439us) client_negotiation.cc:769] Sending connection context
0926 02:20:46.050541 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 02:20:46.051030 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":302,"threads_started":1}
I20260926 02:20:46.054250 21806 heartbeater.cc:347] Connected to a master server at 127.12.244.254:41137
I20260926 02:20:46.054608 21806 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:46.055485 21806 heartbeater.cc:511] Master 127.12.244.254:41137 requested a full tablet report, sending...
I20260926 02:20:46.057914 21615 ts_manager.cc:194] Registered new tserver with Master: 56908119d2b5428193e2862a08bc13bc (127.12.244.193:45109)
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:20:46.059677 21615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:49981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:20:46.090963 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:41137
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:46.438604 21812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:46.438967 21812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:46.439167 21812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:46.446615 21812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:46.446798 21812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:46.446997 21812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:46.457734 21812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:46.459228 21812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:46.460618 21812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:46.474866 21817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:46.474794 21820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:46.474888 21818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:46.475608 21812 server_base.cc:1034] running on GCE node
I20260926 02:20:46.476260 21812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:46.477391 21812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:46.478641 21812 hybrid_clock.cc:648] HybridClock initialized: now 1790389246478567 us; error 105 us; skew 500 ppm
I20260926 02:20:46.478994 21812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:46 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389246, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:46.482429 21812 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:20:46.484261 21812 webserver.cc:460] Webserver started at http://127.12.244.194:39081/ using document root <none> and password file <none>
I20260926 02:20:46.485008 21812 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:46.485157 21812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:46.485486 21812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:46.488554 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "951b4fa1ce2546bb934fe3900874b401"
format_stamp: "Formatted at 2026-09-26 02:20:46 on dist-test-slave-qfmw"
server_key: "9ebbeb9d5a96f28ebeb5b7bd295e1eb9"
server_key_iv: "cf40db6b1864e4a81a4deefc51ef9c3e"
server_key_version: "encryptionkey@0"
I20260926 02:20:46.489370 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "951b4fa1ce2546bb934fe3900874b401"
format_stamp: "Formatted at 2026-09-26 02:20:46 on dist-test-slave-qfmw"
server_key: "9ebbeb9d5a96f28ebeb5b7bd295e1eb9"
server_key_iv: "cf40db6b1864e4a81a4deefc51ef9c3e"
server_key_version: "encryptionkey@0"
I20260926 02:20:46.494887 21812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 02:20:46.499411 21827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:46.500972 21812 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:46.501165 21812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "951b4fa1ce2546bb934fe3900874b401"
format_stamp: "Formatted at 2026-09-26 02:20:46 on dist-test-slave-qfmw"
server_key: "9ebbeb9d5a96f28ebeb5b7bd295e1eb9"
server_key_iv: "cf40db6b1864e4a81a4deefc51ef9c3e"
server_key_version: "encryptionkey@0"
I20260926 02:20:46.501415 21812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:46.525283 21812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:46.530071 21812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:46.530416 21812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:46.531437 21812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:46.532797 21812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:46.532907 21812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:46.533059 21812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:46.533133 21812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:46.568022 21812 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:46191
I20260926 02:20:46.568071 21940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:46191 every 8 connection(s)
I20260926 02:20:46.569424 21812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:46.575158 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 21812
I20260926 02:20:46.575331 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:20:46.575778 13267 external_mini_cluster.cc:1428] Setting key b491c1b770bcd8a4949f9d9703743493
Sep 26 02:20:46 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389246, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:46.592110 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:46.572056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:47883 (local address 127.12.244.254:41137)
0926 02:20:46.572491 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:46.572496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:46.573543 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 02:20:46.575506 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:46.575512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:46.575614 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:46.575730 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:46.577752 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.578796 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.580499 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.581239 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.581385 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:46.585320 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:46.585335 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:46.585342 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:46.585390 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:46.588382 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:46.589436 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:46.589440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:46.589442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:46.589574 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:46.590124 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:46.590128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:46.590131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:46.590450 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:46.590581 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:46.591605 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0926 02:20:46.591844 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260926 02:20:46.593698 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:46.572681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:47883)
0926 02:20:46.573378 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:46.573422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:46.575161 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:46.575996 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:46.576015 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:46.576705 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:46.577563 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:46.577596 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.578997 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.579002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:46.580348 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:46.580360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:46.581464 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:46.581469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:46.581687 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:46.582438 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:46.582463 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:46.585043 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:46.588556 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:46.588569 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:46.588591 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:46.589262 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:46.589694 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:46.589701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:46.589707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:46.589971 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:46.590597 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:46.590609 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:46.591157 (+ 548us) client_negotiation.cc:769] Sending connection context
0926 02:20:46.591487 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 02:20:46.592095 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":268,"threads_started":1}
I20260926 02:20:46.595464 21941 heartbeater.cc:347] Connected to a master server at 127.12.244.254:41137
I20260926 02:20:46.595928 21941 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:46.596788 21941 heartbeater.cc:511] Master 127.12.244.254:41137 requested a full tablet report, sending...
I20260926 02:20:46.598658 21615 ts_manager.cc:194] Registered new tserver with Master: 951b4fa1ce2546bb934fe3900874b401 (127.12.244.194:46191)
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
I20260926 02:20:46.599733 21615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:47883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:20:46.630857 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:41137
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:46.973685 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:46.974061 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:46.974226 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:46.981634 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:46.981817 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:46.982008 21947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:46.992129 21947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:46.993569 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:46.995036 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:47.009492 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:47.009533 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:47.010214 21947 server_base.cc:1034] running on GCE node
W20260926 02:20:47.010295 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:47.011000 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:47.012162 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:47.013432 21947 hybrid_clock.cc:648] HybridClock initialized: now 1790389247013367 us; error 76 us; skew 500 ppm
I20260926 02:20:47.013783 21947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:47 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389247, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:47.017261 21947 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:20:47.018778 21947 webserver.cc:460] Webserver started at http://127.12.244.195:45283/ using document root <none> and password file <none>
I20260926 02:20:47.019603 21947 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:47.019713 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:47.019987 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:20:47.023335 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "e19f6193d87e4361974819396abc82ab"
format_stamp: "Formatted at 2026-09-26 02:20:47 on dist-test-slave-qfmw"
server_key: "ef73a22303caec1373a0cf67393d9980"
server_key_iv: "ef05b22df632d54ad950c96b758667fe"
server_key_version: "encryptionkey@0"
I20260926 02:20:47.024134 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "e19f6193d87e4361974819396abc82ab"
format_stamp: "Formatted at 2026-09-26 02:20:47 on dist-test-slave-qfmw"
server_key: "ef73a22303caec1373a0cf67393d9980"
server_key_iv: "ef05b22df632d54ad950c96b758667fe"
server_key_version: "encryptionkey@0"
I20260926 02:20:47.030083 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:20:47.034034 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:47.035684 21947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:20:47.035897 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "e19f6193d87e4361974819396abc82ab"
format_stamp: "Formatted at 2026-09-26 02:20:47 on dist-test-slave-qfmw"
server_key: "ef73a22303caec1373a0cf67393d9980"
server_key_iv: "ef05b22df632d54ad950c96b758667fe"
server_key_version: "encryptionkey@0"
I20260926 02:20:47.036126 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:47.056907 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:47.061726 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:47.062067 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:47.063257 21947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:47.063604 21806 heartbeater.cc:503] Master 127.12.244.254:41137 was elected leader, sending a full tablet report...
I20260926 02:20:47.064944 21947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:47.065085 21947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 02:20:47.065258 21947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:47.065359 21947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:47.101598 21947 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:34417
I20260926 02:20:47.101656 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:34417 every 8 connection(s)
I20260926 02:20:47.102921 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:47.104216 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 21947
I20260926 02:20:47.104393 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:20:47.104960 13267 external_mini_cluster.cc:1428] Setting key c559880929e0c639598ae54d1317b3aa
Sep 26 02:20:47 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389247, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:20:47.124781 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:47.105947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:53229 (local address 127.12.244.254:41137)
0926 02:20:47.106441 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:47.106447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:47.107973 (+ 1526us) server_negotiation.cc:407] Connection header received
0926 02:20:47.109859 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:47.109863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:47.109928 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:47.110017 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:47.111662 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.112419 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.113997 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.114407 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.114507 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:47.118235 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:47.118240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:47.118243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:47.118302 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:47.121425 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.122322 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.122333 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.122336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.122464 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.122921 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.122932 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.122934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.123265 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:47.123423 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:47.124167 (+ 744us) server_negotiation.cc:299] Negotiation successful
0926 02:20:47.124391 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":229,"threads_started":1}
I20260926 02:20:47.125792 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:47.106753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:53229)
0926 02:20:47.107798 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:47.107853 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:47.109610 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:47.110243 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:47.110251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:47.110801 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:47.111502 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:47.111520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.112560 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.112564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:47.113831 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:47.113843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.114618 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.114623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:47.114844 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:47.115614 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:47.115643 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:47.117983 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:20:47.121579 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:47.121589 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:47.121610 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:47.122170 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:47.122580 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:20:47.122586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:20:47.122590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:20:47.122802 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:20:47.123393 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:20:47.123402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:20:47.123801 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 02:20:47.124082 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 02:20:47.124670 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":388,"threads_started":1}
I20260926 02:20:47.127264 22077 heartbeater.cc:347] Connected to a master server at 127.12.244.254:41137
I20260926 02:20:47.127609 22077 heartbeater.cc:464] Registering TS with master...
I20260926 02:20:47.128393 22077 heartbeater.cc:511] Master 127.12.244.254:41137 requested a full tablet report, sending...
I20260926 02:20:47.130012 21615 ts_manager.cc:194] Registered new tserver with Master: e19f6193d87e4361974819396abc82ab (127.12.244.195:34417)
I20260926 02:20:47.130858 21615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:53229
I20260926 02:20:47.133636 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab.
Entry for principal oryx/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab.
WARNING: no policy specified for oryx/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab.
Entry for principal oryx/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab.
WARNING: no policy specified for oryx/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab.
Entry for principal oryx/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab.
I20260926 02:20:47.282572 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 21676
Sep 26 02:20:47 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:20:47.300807 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:47.283775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.12.244.193:45109)
0926 02:20:47.284433 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:47.284443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:47.284478 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:20:47.284665 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:47.284672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:47.284904 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:47.285079 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:47.286578 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.287714 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.289032 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.289639 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.289791 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:47.294308 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:47.294325 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:47.294342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:47.294384 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:47.297494 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.298409 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.298417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.298419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.298565 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.299040 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.299043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.299046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.299588 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:47.299852 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:47.300146 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 02:20:47.300383 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":295,"threads_started":1}
I20260926 02:20:47.302083 21801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:47.389675 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 21676
I20260926 02:20:47.407418 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 21812
Sep 26 02:20:47 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:20:47.425290 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:47.408364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.12.244.194:46191)
0926 02:20:47.408918 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:47.408925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:47.408952 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:20:47.409011 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:47.409019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:47.409272 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:47.409414 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:47.410753 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.411933 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.413292 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.413973 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.414097 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:47.418825 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:47.418847 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:47.418865 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:47.418915 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:47.422143 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.422957 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.422965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.422967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.423156 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.423562 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.423568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.423571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.423953 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:47.424264 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:47.424618 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 02:20:47.424894 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
I20260926 02:20:47.426703 21936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:47.513288 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 21812
I20260926 02:20:47.535189 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 21947
Sep 26 02:20:47 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:20:47.559043 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:47.536555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.12.244.195:34417)
0926 02:20:47.537227 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:47.537235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:47.537286 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:20:47.537392 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:47.537400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:47.537648 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:47.537926 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:47.540066 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.541419 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.543761 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:47.544424 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:47.544559 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:47.550130 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:47.550148 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:47.550166 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:47.550223 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:47.554357 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.555597 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.555607 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.555610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.555804 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:47.556776 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:47.556780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:47.556783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:47.557287 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:47.557765 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:47.558110 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 02:20:47.558496 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":340,"threads_started":1}
I20260926 02:20:47.560853 22072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:47.645906 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 21947
I20260926 02:20:47.661902 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 21583
I20260926 02:20:47.663221 21645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:20:47.778612 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 21583
I20260926 02:20:47.799827 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:41137
--webserver_interface=127.12.244.254
--webserver_port=32957
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:41137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:20:48.145107 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:48.145427 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:48.145562 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:48.153033 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:20:48.153162 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:48.153254 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:48.153366 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:20:48.153431 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:20:48.163558 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:41137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:41137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=32957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:48.165001 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:48.166421 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:48.178927 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:48.178988 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:48.178964 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:48.179905 22095 server_base.cc:1034] running on GCE node
I20260926 02:20:48.180820 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:48.182472 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:48.183723 22095 hybrid_clock.cc:648] HybridClock initialized: now 1790389248183653 us; error 70 us; skew 500 ppm
I20260926 02:20:48.184147 22095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389248, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:48.187402 22095 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:20:48.189003 22095 webserver.cc:460] Webserver started at http://127.12.244.254:32957/ using document root <none> and password file <none>
I20260926 02:20:48.189700 22095 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:48.189824 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:48.195849 22095 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:48.198992 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:48.200558 22095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:20:48.200834 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "2c7f2f2f8cdd42029915164b9834bb16"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "86c563959d06e1328c789b055efc5e18"
server_key_iv: "af33f5856e2e2a958b40ba6e813f689a"
server_key_version: "encryptionkey@0"
I20260926 02:20:48.201843 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:48.233160 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:48.238178 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:48.238524 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:48.258275 22095 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:41137
I20260926 02:20:48.258298 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:41137 every 8 connection(s)
I20260926 02:20:48.259986 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:20:48.263643 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22095
I20260926 02:20:48.264699 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:45109
--local_ip_for_outbound_sockets=127.12.244.193
--tserver_master_addrs=127.12.244.254:41137
--webserver_port=38905
--webserver_interface=127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:20:48.269663 22162 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:20:48.274381 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Bootstrap starting.
I20260926 02:20:48.294193 22162 log.cc:824] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Log is configured to *not* fsync() on all Append() calls
I20260926 02:20:48.303820 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:20:48.304388 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Bootstrap complete.
I20260926 02:20:48.314043 22162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:48.314939 22162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c7f2f2f8cdd42029915164b9834bb16, State: Initialized, Role: FOLLOWER
I20260926 02:20:48.315675 22162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:48.315923 22162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:20:48.316042 22162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:20:48.316248 22162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:20:48.320389 22162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:48.320986 22162 leader_election.cc:302] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c7f2f2f8cdd42029915164b9834bb16; no voters:
I20260926 02:20:48.321659 22162 leader_election.cc:288] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:20:48.321812 22167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:20:48.323200 22167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [term 2 LEADER]: Becoming Leader. State: Replica: 2c7f2f2f8cdd42029915164b9834bb16, State: Running, Role: LEADER
I20260926 02:20:48.323877 22167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } }
I20260926 02:20:48.324196 22162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:20:48.326908 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c7f2f2f8cdd42029915164b9834bb16. Latest consensus state: current_term: 2 leader_uuid: "2c7f2f2f8cdd42029915164b9834bb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } } }
I20260926 02:20:48.327363 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:48.332166 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c7f2f2f8cdd42029915164b9834bb16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c7f2f2f8cdd42029915164b9834bb16" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 41137 } } }
I20260926 02:20:48.332602 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16 [sys.catalog]: This master's current role is: LEADER
I20260926 02:20:48.337085 22174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:20:48.344296 22174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:20:48.345976 22174 catalog_manager.cc:1259] Loaded cluster ID: 5ccbf071c6504f37b826fa36f11cc549
I20260926 02:20:48.346194 22174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:20:48.350622 22174 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:20:48.352517 22174 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c7f2f2f8cdd42029915164b9834bb16: Loaded TSK: 0
I20260926 02:20:48.354455 22174 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:20:48.647462 22165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:48.647761 22165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:48.647905 22165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:48.655710 22165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:48.655839 22165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:48.656008 22165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:20:48.666441 22165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.193.keytab
--principal=oryx/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:45109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=38905
--tserver_master_addrs=127.12.244.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:48.667949 22165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:48.669432 22165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:48.685159 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:48.685127 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:48.685206 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:48.686856 22165 server_base.cc:1034] running on GCE node
I20260926 02:20:48.687749 22165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:48.688865 22165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:48.690145 22165 hybrid_clock.cc:648] HybridClock initialized: now 1790389248690065 us; error 49 us; skew 500 ppm
I20260926 02:20:48.690524 22165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389248, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:48.693967 22165 init.cc:377] Logged in from keytab as oryx/127.12.244.193@KRBTEST.COM (short username oryx)
I20260926 02:20:48.705427 22165 webserver.cc:460] Webserver started at http://127.12.244.193:38905/ using document root <none> and password file <none>
I20260926 02:20:48.706534 22165 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:48.706704 22165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:48.715219 22165 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:20:48.718525 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:48.720148 22165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:20:48.720363 22165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "56908119d2b5428193e2862a08bc13bc"
format_stamp: "Formatted at 2026-09-26 02:20:45 on dist-test-slave-qfmw"
server_key: "0caa1a32bd8c41b92467e23fa0275c8d"
server_key_iv: "8ea616c54f6f08abb42e29041b3db00d"
server_key_version: "encryptionkey@0"
I20260926 02:20:48.721383 22165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:48.738281 22165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:48.742941 22165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:48.743257 22165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:48.744465 22165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:48.745857 22165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:48.745944 22165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:48.746106 22165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:48.746200 22165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:48.781510 22165 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45109
I20260926 02:20:48.781620 22312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45109 every 8 connection(s)
I20260926 02:20:48.783156 22165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:20:48.785318 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22165
I20260926 02:20:48.786259 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:46191
--local_ip_for_outbound_sockets=127.12.244.194
--tserver_master_addrs=127.12.244.254:41137
--webserver_port=39081
--webserver_interface=127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:20:48.806325 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:48.786687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:49043 (local address 127.12.244.254:41137)
0926 02:20:48.787553 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:48.787573 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:48.788170 (+ 597us) server_negotiation.cc:407] Connection header received
0926 02:20:48.790509 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:48.790534 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:48.791178 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:48.791629 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:48.793101 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.794807 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.797581 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.798389 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.798553 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:48.805206 (+ 6653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:49043: BlockingRecv error: recv got EOF from 127.12.244.193:49043 (error 108)
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":342,"threads_started":1}
I20260926 02:20:48.806296 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:48.786689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:49043)
0926 02:20:48.787957 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:48.788014 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:48.789846 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:48.791738 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:48.791746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:48.792294 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:48.792879 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:48.792898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.794996 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.795005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:48.797321 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:48.797344 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.798684 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.798694 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:48.799044 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:48.799939 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:48.799980 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:48.804727 (+ 4747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":751,"threads_started":1}
W20260926 02:20:48.807070 22313 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:41137 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.244.254:41137: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:48.823681 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:48.807771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:42957)
0926 02:20:48.808015 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:48.808041 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:48.808177 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:48.808521 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:48.808528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:48.809031 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:48.809276 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:48.809291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.810312 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.810319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:48.812177 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:48.812198 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.813027 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.813036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:48.813219 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:48.820026 (+ 6807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:48.820056 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:48.823456 (+ 3400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:20:48.823997 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:48.807964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:42957 (local address 127.12.244.254:41137)
0926 02:20:48.808133 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:48.808137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:48.808155 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:20:48.808308 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:48.808312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:48.808372 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:48.808453 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:48.809434 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.810154 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.812383 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.812850 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.812979 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:48.823764 (+ 10785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:42957: BlockingRecv error: recv got EOF from 127.12.244.193:42957 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:48 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:48.833434 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:48.824600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:43709)
0926 02:20:48.824776 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:48.824795 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:48.824956 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:48.825325 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:48.825328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:48.825799 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:48.826028 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:48.826041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.827456 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.827461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:48.828511 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:48.828525 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.829426 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.829431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:48.829583 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:48.830261 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:48.830278 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:48.833202 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 02:20:48.833737 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:48.824796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:43709 (local address 127.12.244.254:41137)
0926 02:20:48.824995 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:48.825001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:48.825021 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:20:48.825083 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:48.825087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:48.825176 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:48.825280 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:48.826188 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.827312 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.828663 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:48.829273 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:48.829398 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:48.833477 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:43709: BlockingRecv error: recv got EOF from 127.12.244.193:43709 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 02:20:48.833843 22313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:20:49.165802 22317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:49.166117 22317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:49.166216 22317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:49.174093 22317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:49.174226 22317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:49.174350 22317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:20:49.184691 22317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.194.keytab
--principal=oryx/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:46191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=39081
--tserver_master_addrs=127.12.244.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:49.186185 22317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:49.187659 22317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:49.202157 22323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:49.202206 22325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:49.202163 22322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:49.203194 22317 server_base.cc:1034] running on GCE node
I20260926 02:20:49.203882 22317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:49.204947 22317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:49.206197 22317 hybrid_clock.cc:648] HybridClock initialized: now 1790389249206131 us; error 53 us; skew 500 ppm
I20260926 02:20:49.206606 22317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389249, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:49.210095 22317 init.cc:377] Logged in from keytab as oryx/127.12.244.194@KRBTEST.COM (short username oryx)
I20260926 02:20:49.211817 22317 webserver.cc:460] Webserver started at http://127.12.244.194:39081/ using document root <none> and password file <none>
I20260926 02:20:49.212634 22317 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:49.212754 22317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:49.218652 22317 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 02:20:49.221589 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:49.223433 22317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:20:49.223605 22317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "951b4fa1ce2546bb934fe3900874b401"
format_stamp: "Formatted at 2026-09-26 02:20:46 on dist-test-slave-qfmw"
server_key: "9ebbeb9d5a96f28ebeb5b7bd295e1eb9"
server_key_iv: "cf40db6b1864e4a81a4deefc51ef9c3e"
server_key_version: "encryptionkey@0"
I20260926 02:20:49.224888 22317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:49.259479 22317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:49.265377 22317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:49.265774 22317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:49.267028 22317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:49.268672 22317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:49.268770 22317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:49.268927 22317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:49.269021 22317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:49.308121 22317 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:46191
I20260926 02:20:49.308229 22444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:46191 every 8 connection(s)
I20260926 02:20:49.309864 22317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:20:49.315021 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22317
I20260926 02:20:49.315969 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:34417
--local_ip_for_outbound_sockets=127.12.244.195
--tserver_master_addrs=127.12.244.254:41137
--webserver_port=45283
--webserver_interface=127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 02:20:49.334335 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:49.313374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:59823 (local address 127.12.244.254:41137)
0926 02:20:49.313618 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.313624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.315494 (+ 1870us) server_negotiation.cc:407] Connection header received
0926 02:20:49.319554 (+ 4060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.319559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.319741 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.319871 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.322916 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.323960 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.326537 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.327535 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.327743 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.333994 (+ 6251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:59823: BlockingRecv error: recv got EOF from 127.12.244.194:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:20:49.334983 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.313909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:59823)
0926 02:20:49.315239 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:49.315312 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:49.318998 (+ 3686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:49.320285 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:49.320303 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:49.321374 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:49.322558 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.322597 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.324200 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.324209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.326270 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.326288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.327829 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.327835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.328088 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.328918 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:49.328961 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:49.333629 (+ 4668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":460,"threads_started":1}
W20260926 02:20:49.335750 22445 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:41137 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.244.254:41137: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:49.347764 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.336414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:36449)
0926 02:20:49.336643 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:49.336667 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:49.336801 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:49.337265 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:49.337273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:49.337777 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:49.338023 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.338038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.339207 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.339214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.341097 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.341130 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.342176 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.342183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.342445 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.343523 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:49.343554 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:49.347465 (+ 3911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:20:49.348119 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:49.336553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:36449 (local address 127.12.244.254:41137)
0926 02:20:49.336722 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.336728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.336751 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:20:49.336952 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.336959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.337089 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.337204 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.338282 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.339023 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.341438 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.341941 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.342083 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.347708 (+ 5625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:36449: BlockingRecv error: recv got EOF from 127.12.244.194:36449 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:49.357609 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.348592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:42031)
0926 02:20:49.348791 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:49.348810 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:49.348921 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:49.349268 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:49.349273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:49.349658 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:49.349855 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.349866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.350946 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.350956 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.351996 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.352010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.352743 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.352750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.352954 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.353641 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:49.353661 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:49.357186 (+ 3525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 02:20:49.357677 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:49.348738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:42031 (local address 127.12.244.254:41137)
0926 02:20:49.348894 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.348899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.348916 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:20:49.349037 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.349040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.349113 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.349195 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.350025 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.350781 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.352166 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.352579 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.352726 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.357450 (+ 4724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:42031: BlockingRecv error: recv got EOF from 127.12.244.194:42031 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 02:20:49.357975 22445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:20:49.716265 22448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:20:49.716601 22448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:20:49.716701 22448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:20:49.726020 22448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:20:49.726218 22448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:20:49.726374 22448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:20:49.739010 22448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/krb5kdc/oryx_127.12.244.195.keytab
--principal=oryx/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=45283
--tserver_master_addrs=127.12.244.254:41137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:20:49.740674 22448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:20:49.742779 22448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:20:49.758220 22453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:49.758311 22454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:20:49.758244 22456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:20:49.759475 22448 server_base.cc:1034] running on GCE node
I20260926 02:20:49.760250 22448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:20:49.761554 22448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:20:49.762879 22448 hybrid_clock.cc:648] HybridClock initialized: now 1790389249762852 us; error 56 us; skew 500 ppm
I20260926 02:20:49.763413 22448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389249, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:20:49.767000 22448 init.cc:377] Logged in from keytab as oryx/127.12.244.195@KRBTEST.COM (short username oryx)
I20260926 02:20:49.768885 22448 webserver.cc:460] Webserver started at http://127.12.244.195:45283/ using document root <none> and password file <none>
I20260926 02:20:49.769701 22448 fs_manager.cc:359] Metadata directory not provided
I20260926 02:20:49.769856 22448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:20:49.776293 22448 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:20:49.780298 22463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:49.782474 22448 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:20:49.782686 22448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "e19f6193d87e4361974819396abc82ab"
format_stamp: "Formatted at 2026-09-26 02:20:47 on dist-test-slave-qfmw"
server_key: "ef73a22303caec1373a0cf67393d9980"
server_key_iv: "ef05b22df632d54ad950c96b758667fe"
server_key_version: "encryptionkey@0"
I20260926 02:20:49.785532 22448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:20:49.808426 22448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:20:49.813369 22448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:20:49.813608 22448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:20:49.814682 22448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:20:49.816150 22448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:20:49.816257 22448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:20:49.816351 22448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:20:49.816411 22448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:49.847500 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.834615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:46319)
0926 02:20:49.835310 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:49.835382 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:49.835571 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:49.836063 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:49.836071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:49.836689 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:49.837003 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.837018 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.838538 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.838546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.840127 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.840144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.841316 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.841327 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.841599 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.842444 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:49.842480 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:49.847103 (+ 4623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":448,"threads_started":1}
W20260926 02:20:49.847692 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:49.834760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:46319 (local address 127.12.244.254:41137)
0926 02:20:49.834994 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.835003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.835511 (+ 508us) server_negotiation.cc:407] Connection header received
0926 02:20:49.835733 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.835741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.835855 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.835997 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.837194 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.838380 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.840381 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.841033 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.841218 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.847416 (+ 6198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:46319: BlockingRecv error: recv got EOF from 127.12.244.193:46319 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:20:49.856534 22448 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:34417
I20260926 02:20:49.856638 22577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:34417 every 8 connection(s)
I20260926 02:20:49.858178 22448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:20:49.858587 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22448
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:49.878170 22315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.860306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.12.244.254:41137)
0926 02:20:49.860529 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.860535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.860562 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:20:49.860639 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.860645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.860731 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.860873 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.862538 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.863589 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.865516 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.866221 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.866327 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.869821 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:20:49.869831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:20:49.869849 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:20:49.869899 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:20:49.873886 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:49.875300 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:49.875313 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:49.875327 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:49.875556 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:20:49.876013 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:20:49.876018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:20:49.876023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:20:49.876576 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:20:49.876904 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 02:20:49.877401 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 02:20:49.877845 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:20:49.878449 22582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:49.862182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:55705 (local address 127.12.244.254:41137)
0926 02:20:49.863452 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.863458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.863478 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:20:49.865519 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.865523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.865628 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.865755 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.868206 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.869046 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.871277 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.872090 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.872247 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.878088 (+ 5841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:55705: BlockingRecv error: recv got EOF from 127.12.244.195:55705 (error 108)
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":1097,"thread_start_us":1073,"threads_started":1}
I20260926 02:20:49.879036 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.861784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:55705)
0926 02:20:49.862662 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:49.862706 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:49.865139 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:49.866047 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:49.866058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:49.866939 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:49.867942 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.867978 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.869270 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.869277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.871018 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.871033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.872500 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.872509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.872748 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.873449 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:49.873479 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:49.877649 (+ 4170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":438,"threads_started":1}
W20260926 02:20:49.879740 22578 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:41137 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.244.254:41137: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:49.888659 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.880353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:52959)
0926 02:20:49.880716 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:49.880736 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:49.880880 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:49.881241 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:49.881245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:49.881606 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:49.881805 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.881816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.883005 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.883009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.884037 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.884047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.884730 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.884734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.884870 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.885436 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:49.885453 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:49.888452 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248}
W20260926 02:20:49.889024 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:49.880447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:52959 (local address 127.12.244.254:41137)
0926 02:20:49.880694 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.880701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.880815 (+ 114us) server_negotiation.cc:407] Connection header received
0926 02:20:49.881032 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.881037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.881109 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.881215 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.881955 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.882875 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.884164 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.884602 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.884698 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.888682 (+ 3984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:52959: BlockingRecv error: recv got EOF from 127.12.244.195:52959 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:49 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:49.898311 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:49.889402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:52165)
0926 02:20:49.889571 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:49.889590 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:49.889689 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:49.889963 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:49.889966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:49.890238 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:49.890386 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.890393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.891444 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.891447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.892564 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:49.892576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.893469 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.893473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:49.893786 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.894574 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:49.894607 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:49.898054 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:20:49.898644 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:49.889514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:52165 (local address 127.12.244.254:41137)
0926 02:20:49.889657 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:49.889661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:49.889678 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:20:49.889776 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:49.889779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:49.889839 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:49.889928 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:49.890518 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.891300 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.892720 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:49.893265 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:49.893429 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:49.898335 (+ 4906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:52165: BlockingRecv error: recv got EOF from 127.12.244.195:52165 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 02:20:49.898707 22578 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 02:20:50 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:50 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:50.374701 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:50.358864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:45349)
0926 02:20:50.359426 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:50.359448 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:50.359597 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:50.360022 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:50.360030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:50.360702 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:50.360912 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:50.360922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.362072 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.362077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:50.363160 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:50.363183 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.364127 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.364132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:50.364270 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:50.370921 (+ 6651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:50.370945 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:50.374464 (+ 3519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":396,"threads_started":1}
W20260926 02:20:50.374933 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:50.358991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:45349 (local address 127.12.244.254:41137)
0926 02:20:50.359259 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:50.359265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:50.359567 (+ 302us) server_negotiation.cc:407] Connection header received
0926 02:20:50.359731 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:50.359735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:50.359834 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:50.359939 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:50.361109 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.361901 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.363348 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.363894 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.364030 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:50.374694 (+ 10664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:45349: BlockingRecv error: recv got EOF from 127.12.244.194:45349 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 02:20:50 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:50 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:50.858036 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:50.848795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:41067)
0926 02:20:50.849369 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:50.849390 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:50.849516 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:50.849898 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:50.849901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:50.850289 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:50.850459 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:50.850468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.851966 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.851972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:50.852994 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:50.853008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.853827 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.853832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:50.853953 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:50.854587 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:50.854605 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:50.857660 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":319,"threads_started":1}
W20260926 02:20:50.858119 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:50.848945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:41067 (local address 127.12.244.254:41137)
0926 02:20:50.849150 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:50.849154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:50.849483 (+ 329us) server_negotiation.cc:407] Connection header received
0926 02:20:50.849630 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:50.849635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:50.849699 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:50.849851 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:50.850596 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.851728 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.853137 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.853585 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.853686 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:50.857879 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:41067: BlockingRecv error: recv got EOF from 127.12.244.193:41067 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 02:20:50 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:50 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:50.908442 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:50.899535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:48193)
0926 02:20:50.900119 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:50.900136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:50.900251 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:50.900597 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:50.900601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:50.901236 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:50.901481 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:50.901494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.902590 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.902593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:50.903666 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:50.903678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.904370 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.904375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:50.904491 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:50.905106 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:50.905124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:50.908211 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":343,"threads_started":1}
W20260926 02:20:50.908645 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:50.899645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:48193 (local address 127.12.244.254:41137)
0926 02:20:50.899851 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:50.899857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:50.900223 (+ 366us) server_negotiation.cc:407] Connection header received
0926 02:20:50.900352 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:50.900357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:50.900437 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:50.900517 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:50.901635 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.902466 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.903790 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:50.904191 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:50.904294 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:50.908415 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:48193: BlockingRecv error: recv got EOF from 127.12.244.195:48193 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:20:51 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:51 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:51.384364 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:51.375897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:58197)
0926 02:20:51.376445 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:51.376465 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:51.376606 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:51.377047 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:51.377051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:51.377460 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:51.377621 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:51.377629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.378616 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.378622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:51.379542 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:51.379553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.380321 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.380327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:51.380496 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:51.381095 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:51.381112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:51.383990 (+ 2878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":304,"threads_started":1}
W20260926 02:20:51.384460 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:51.376135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:58197 (local address 127.12.244.254:41137)
0926 02:20:51.376329 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:51.376335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:51.376578 (+ 243us) server_negotiation.cc:407] Connection header received
0926 02:20:51.376780 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:51.376787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:51.376900 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:51.377013 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:51.377774 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.378475 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.379687 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.380154 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.380294 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:51.384225 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:58197: BlockingRecv error: recv got EOF from 127.12.244.194:58197 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 02:20:51 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:51 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:51.868445 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:51.859031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:46665)
0926 02:20:51.859834 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:51.859856 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:51.860016 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:51.860624 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:51.860628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:51.861108 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:51.861292 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:51.861301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.862397 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.862400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:51.863407 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:51.863419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.864197 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.864200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:51.864326 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:51.864958 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:51.864976 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:51.868149 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":540,"threads_started":1}
W20260926 02:20:51.868649 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:51.859271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:46665 (local address 127.12.244.254:41137)
0926 02:20:51.859447 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:51.859455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:51.859951 (+ 496us) server_negotiation.cc:407] Connection header received
0926 02:20:51.860226 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:51.860232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:51.860319 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:51.860448 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:51.861461 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.862256 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.863572 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.864052 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.864182 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:51.868393 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:46665: BlockingRecv error: recv got EOF from 127.12.244.193:46665 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 02:20:51 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:51 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:51.918345 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:51.909631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:56737)
0926 02:20:51.910181 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:51.910196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:51.910362 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:51.910694 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:51.910699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:51.911160 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:51.911352 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:51.911361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.912397 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.912405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:51.913312 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:51.913323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.914093 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.914099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:51.914271 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:51.914952 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:51.914971 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:51.918084 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":289,"threads_started":1}
W20260926 02:20:51.918656 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:51.909787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:56737 (local address 127.12.244.254:41137)
0926 02:20:51.910003 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:51.910009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:51.910285 (+ 276us) server_negotiation.cc:407] Connection header received
0926 02:20:51.910485 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:51.910490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:51.910560 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:51.910663 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:51.911492 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.912232 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.913458 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:51.913916 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:51.914056 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:51.918394 (+ 4338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:56737: BlockingRecv error: recv got EOF from 127.12.244.195:56737 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 02:20:52 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:52 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:52.394135 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:52.385305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:46489)
0926 02:20:52.385881 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:52.385897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:52.386017 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:52.386379 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:52.386382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:52.386713 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:52.386887 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:52.386897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.388022 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.388026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:52.388929 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:52.388940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.389620 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.389624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:52.389746 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:52.390462 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:52.390480 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:52.393737 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":321,"threads_started":1}
W20260926 02:20:52.394228 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:52.385449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:46489 (local address 127.12.244.254:41137)
0926 02:20:52.385616 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:52.385620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:52.385981 (+ 361us) server_negotiation.cc:407] Connection header received
0926 02:20:52.386155 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:52.386159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:52.386228 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:52.386322 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:52.387048 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.387843 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.389067 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.389489 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.389591 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:52.393965 (+ 4374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:46489: BlockingRecv error: recv got EOF from 127.12.244.194:46489 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 02:20:52 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:52 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:52.878871 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:52.869693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:35473)
0926 02:20:52.870184 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:52.870205 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:52.870369 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:52.870796 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:52.870800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:52.871203 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:52.871352 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:52.871360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.872721 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.872729 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:52.873724 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:52.873737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.874624 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.874629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:52.874768 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:52.875455 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:52.875474 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:52.878630 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":273,"threads_started":1}
W20260926 02:20:52.879254 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:52.869906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:35473 (local address 127.12.244.254:41137)
0926 02:20:52.870155 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:52.870163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:52.870348 (+ 185us) server_negotiation.cc:407] Connection header received
0926 02:20:52.870495 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:52.870500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:52.870585 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:52.870690 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:52.871500 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.872517 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.873980 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.874436 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.874556 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:52.878957 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:35473: BlockingRecv error: recv got EOF from 127.12.244.193:35473 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 26 02:20:52 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:52 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:52.929004 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:52.919493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:52063)
0926 02:20:52.920062 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:52.920081 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:52.920225 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:52.920639 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:52.920643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:52.921062 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:52.921231 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:52.921240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.922367 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.922402 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:52.923511 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:52.923523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.924347 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.924353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:52.924526 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:52.925327 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:52.925356 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:52.928634 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":302,"threads_started":1}
W20260926 02:20:52.929507 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:52.919688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:52063 (local address 127.12.244.254:41137)
0926 02:20:52.919877 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:52.919882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:52.920267 (+ 385us) server_negotiation.cc:407] Connection header received
0926 02:20:52.920356 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:52.920361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:52.920444 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:52.920571 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:52.921406 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.922222 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.923660 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:52.924173 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:52.924326 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:52.929087 (+ 4761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:52063: BlockingRecv error: recv got EOF from 127.12.244.195:52063 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 02:20:53 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:53 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:53.404072 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:53.395377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:56101)
0926 02:20:53.395970 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:53.395986 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:53.396102 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:53.396452 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:53.396454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:53.396851 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:53.396998 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:53.397008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.398031 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.398034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:53.398990 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:53.399000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.399727 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.399732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:53.399899 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:53.400633 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:53.400651 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:53.403837 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":304,"threads_started":1}
W20260926 02:20:53.404281 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:53.395439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:56101 (local address 127.12.244.254:41137)
0926 02:20:53.395609 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:53.395613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:53.396070 (+ 457us) server_negotiation.cc:407] Connection header received
0926 02:20:53.396220 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:53.396224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:53.396315 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:53.396408 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:53.397121 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.397911 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.399144 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.399592 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.399688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:53.404031 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:56101: BlockingRecv error: recv got EOF from 127.12.244.194:56101 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:20:53 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:53 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:53.889134 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:53.880112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:35681)
0926 02:20:53.880659 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:53.880681 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:53.880794 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:53.881174 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:53.881185 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:53.881677 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:53.881918 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:53.881933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.883287 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.883291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:53.884215 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:53.884227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.885041 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.885047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:53.885213 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:53.885849 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:53.885868 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:53.888838 (+ 2970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":283,"threads_started":1}
W20260926 02:20:53.889508 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:53.880283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:35681 (local address 127.12.244.254:41137)
0926 02:20:53.880496 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:53.880502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:53.880778 (+ 276us) server_negotiation.cc:407] Connection header received
0926 02:20:53.880887 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:53.880893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:53.880989 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:53.881090 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:53.882084 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.883085 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.884373 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.884842 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.884969 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:53.889199 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:35681: BlockingRecv error: recv got EOF from 127.12.244.193:35681 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 02:20:53 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:53 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:53.939780 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:53.930204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:60217)
0926 02:20:53.930814 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:53.930833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:53.930966 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:53.931365 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:53.931371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:53.931913 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:53.932164 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:53.932176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.933595 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.933599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:53.934575 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:53.934588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.935467 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.935472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:53.935661 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:53.936298 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:53.936317 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:53.939497 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":313,"threads_started":1}
W20260926 02:20:53.939940 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:53.930265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:60217 (local address 127.12.244.254:41137)
0926 02:20:53.930474 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:53.930478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:53.930943 (+ 465us) server_negotiation.cc:407] Connection header received
0926 02:20:53.931076 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:53.931084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:53.931189 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:53.931295 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:53.932362 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.933443 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.934760 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:53.935288 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:53.935403 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:53.939721 (+ 4318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:60217: BlockingRecv error: recv got EOF from 127.12.244.195:60217 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 02:20:54 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:54 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:54.414238 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:54.405198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:60059)
0926 02:20:54.405821 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:54.405842 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:54.405962 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:54.406294 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:54.406297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:54.406690 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:54.406855 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:54.406864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.408285 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.408290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:54.409245 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:54.409259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.410195 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.410201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:54.410379 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:54.411021 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:54.411044 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:54.413985 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":359,"threads_started":1}
W20260926 02:20:54.414443 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:54.405526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:60059 (local address 127.12.244.254:41137)
0926 02:20:54.405717 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:54.405721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:54.405920 (+ 199us) server_negotiation.cc:407] Connection header received
0926 02:20:54.406066 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:54.406070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:54.406151 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:54.406249 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:54.407190 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.408113 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.409394 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.410001 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.410162 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:54.414178 (+ 4016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:60059: BlockingRecv error: recv got EOF from 127.12.244.194:60059 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:20:54 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:54 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:54.899061 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:54.890297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:51143)
0926 02:20:54.890927 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:54.890948 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:54.891133 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:54.891489 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:54.891515 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:54.891901 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:54.892040 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:54.892048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.893077 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.893080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:54.893968 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:54.893978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.894683 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.894687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:54.894841 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:54.895563 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:54.895580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:54.898755 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":350,"threads_started":1}
W20260926 02:20:54.899277 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:54.890436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:51143 (local address 127.12.244.254:41137)
0926 02:20:54.890621 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:54.890627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:54.891034 (+ 407us) server_negotiation.cc:407] Connection header received
0926 02:20:54.891284 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:54.891288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:54.891352 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:54.891451 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:54.892205 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.892941 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.894100 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.894534 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.894635 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:54.898964 (+ 4329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:51143: BlockingRecv error: recv got EOF from 127.12.244.193:51143 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:20:54 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:54 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:54.949481 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:54.940753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:37347)
0926 02:20:54.941382 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:54.941399 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:54.941558 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:54.941895 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:54.941900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:54.942321 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:54.942480 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:54.942489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.943628 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.943636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:54.944657 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:54.944670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.945389 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.945394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:54.945565 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:54.946223 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:54.946241 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:54.949112 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":293,"threads_started":1}
W20260926 02:20:54.949591 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:54.940947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:37347 (local address 127.12.244.254:41137)
0926 02:20:54.941197 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:54.941205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:54.941491 (+ 286us) server_negotiation.cc:407] Connection header received
0926 02:20:54.941698 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:54.941703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:54.941761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:54.941867 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:54.942634 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.943493 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.944794 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:54.945208 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:54.945367 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:54.949348 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:37347: BlockingRecv error: recv got EOF from 127.12.244.195:37347 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 02:20:55 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:55 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:55.431978 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:55.415411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:45939)
0926 02:20:55.415928 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:55.415945 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:55.416080 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:55.416544 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:55.416548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:55.416956 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:55.417120 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:55.417128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.418188 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.418193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:55.419226 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:55.419241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.420046 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.420050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:55.420183 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:55.427876 (+ 7693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:55.427901 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:55.431738 (+ 3837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":295,"threads_started":1}
W20260926 02:20:55.432227 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:55.415613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:45939 (local address 127.12.244.254:41137)
0926 02:20:55.415880 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:55.415887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:55.416033 (+ 146us) server_negotiation.cc:407] Connection header received
0926 02:20:55.416237 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:55.416242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:55.416346 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:55.416438 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:55.417290 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.418029 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.419391 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.419832 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.419944 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:55.431995 (+ 12051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:45939: BlockingRecv error: recv got EOF from 127.12.244.194:45939 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 26 02:20:55 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:55 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:55.909579 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:55.900022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:37799)
0926 02:20:55.900646 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:55.900665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:55.900844 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:55.901237 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:55.901242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:55.901699 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:55.901866 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:55.901875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.902994 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.902998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:55.904043 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:55.904055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.904868 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.904874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:55.905065 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:55.905894 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:55.905914 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:55.909120 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":303,"threads_started":1}
W20260926 02:20:55.909648 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:55.900132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:37799 (local address 127.12.244.254:41137)
0926 02:20:55.900338 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:55.900346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:55.900768 (+ 422us) server_negotiation.cc:407] Connection header received
0926 02:20:55.901005 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:55.901017 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:55.901084 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:55.901191 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:55.902027 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.902815 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.904209 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.904684 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.904843 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:55.909383 (+ 4540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:37799: BlockingRecv error: recv got EOF from 127.12.244.193:37799 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 02:20:55 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:55 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:55.959821 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:55.950503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:36697)
0926 02:20:55.951241 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:55.951286 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:55.951465 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:55.951879 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:55.951882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:55.952275 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:55.952439 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:55.952447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.953587 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.953591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:55.955132 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:55.955150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.955916 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.955921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:55.956060 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:55.956681 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:55.956700 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:55.959539 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":402,"threads_started":1}
W20260926 02:20:55.960019 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:55.950751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:36697 (local address 127.12.244.254:41137)
0926 02:20:55.950988 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:55.950994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:55.951376 (+ 382us) server_negotiation.cc:407] Connection header received
0926 02:20:55.951580 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:55.951583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:55.951672 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:55.951760 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:55.952620 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.953407 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.955318 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:55.955750 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:55.955912 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:55.959779 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:36697: BlockingRecv error: recv got EOF from 127.12.244.195:36697 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 02:20:56 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:56 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:56.443327 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:56.432975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:43851)
0926 02:20:56.433549 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:56.433565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:56.433678 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:56.434020 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:56.434023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:56.434510 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:56.434679 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:56.434687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.436069 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.436073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:56.437233 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:56.437248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.438205 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.438210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:56.438361 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:56.439217 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:56.439237 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:56.442908 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":271,"threads_started":1}
W20260926 02:20:56.443414 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:56.433025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:43851 (local address 127.12.244.254:41137)
0926 02:20:56.433206 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:56.433210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:56.433641 (+ 431us) server_negotiation.cc:407] Connection header received
0926 02:20:56.433798 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:56.433808 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:56.433884 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:56.433967 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:56.435151 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.435894 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.437412 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.437985 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.438116 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:56.443147 (+ 5031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:43851: BlockingRecv error: recv got EOF from 127.12.244.194:43851 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 02:20:56 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:56 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:56.920333 22602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:56.910769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:42987)
0926 02:20:56.911368 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:56.911390 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:56.911511 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:56.911875 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:56.911879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:56.912271 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:56.912432 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:56.912441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.913540 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.913545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:56.914942 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:56.914956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.915897 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.915905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:56.916086 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:56.916879 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:56.916898 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:56.920090 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":318,"threads_started":1}
W20260926 02:20:56.920756 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:56.910906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:42987 (local address 127.12.244.254:41137)
0926 02:20:56.911173 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:56.911177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:56.911475 (+ 298us) server_negotiation.cc:407] Connection header received
0926 02:20:56.911614 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:56.911618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:56.911684 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:56.911791 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:56.912591 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.913361 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.915121 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.915649 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.915787 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:56.920404 (+ 4617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:42987: BlockingRecv error: recv got EOF from 127.12.244.193:42987 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 02:20:56 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:56 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:56.969930 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:56.960985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:50379)
0926 02:20:56.961642 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:56.961659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:56.961832 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:56.962185 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:56.962189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:56.962605 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:56.962772 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:56.962792 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.963870 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.963873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:56.964800 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:56.964813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.965499 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.965508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:56.965648 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:56.966337 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:56.966361 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:56.969669 (+ 3308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":293,"threads_started":1}
W20260926 02:20:56.970124 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:56.961169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:50379 (local address 127.12.244.254:41137)
0926 02:20:56.961370 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:56.961374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:56.961741 (+ 367us) server_negotiation.cc:407] Connection header received
0926 02:20:56.961942 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:56.961946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:56.962009 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:56.962133 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:56.962934 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.963707 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.964928 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:56.965338 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:56.965435 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:56.969891 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:50379: BlockingRecv error: recv got EOF from 127.12.244.195:50379 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 02:20:57 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:57 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:57.454102 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:57.444471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:49879)
0926 02:20:57.445139 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:57.445156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:57.445323 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:57.445951 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:57.445954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:57.446378 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:57.446551 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:57.446560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.447914 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.447919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:57.448881 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:57.448893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.449613 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.449618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:57.449739 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:57.450480 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:57.450499 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:57.453711 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":370,"threads_started":1}
W20260926 02:20:57.454227 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:57.444656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:49879 (local address 127.12.244.254:41137)
0926 02:20:57.444868 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:57.444875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:57.445247 (+ 372us) server_negotiation.cc:407] Connection header received
0926 02:20:57.445647 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:57.445654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:57.445783 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:57.445911 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:57.446718 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.447733 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.449019 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.449455 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.449552 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:57.453971 (+ 4419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:49879: BlockingRecv error: recv got EOF from 127.12.244.194:49879 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 02:20:57 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:57 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:57.931404 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:57.921356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:48887)
0926 02:20:57.921942 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:57.921965 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:57.922146 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:57.922520 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:57.922525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:57.923017 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:57.923294 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:57.923305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.924505 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.924517 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:57.925740 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:57.925755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.926591 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.926597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:57.926791 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:57.927437 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:57.927457 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:57.930967 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":335,"threads_started":1}
W20260926 02:20:57.931528 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:57.921406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:48887 (local address 127.12.244.254:41137)
0926 02:20:57.921789 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:57.921796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:57.922057 (+ 261us) server_negotiation.cc:407] Connection header received
0926 02:20:57.922279 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:57.922290 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:57.922363 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:57.922487 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:57.923482 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.924334 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.925901 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.926392 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.926571 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:57.931271 (+ 4700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:48887: BlockingRecv error: recv got EOF from 127.12.244.193:48887 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Sep 26 02:20:57 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:57 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:57.979708 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:57.970979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:38505)
0926 02:20:57.971559 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:57.971576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:57.971699 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:57.972045 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:57.972049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:57.972461 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:57.972682 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:57.972693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.973720 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.973725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:57.974662 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:57.974674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.975567 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.975572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:57.975687 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:57.976397 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:57.976414 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:57.979362 (+ 2948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":313,"threads_started":1}
W20260926 02:20:57.979872 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:57.971048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:38505 (local address 127.12.244.254:41137)
0926 02:20:57.971237 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:57.971242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:57.971660 (+ 418us) server_negotiation.cc:407] Connection header received
0926 02:20:57.971818 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:57.971822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:57.971884 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:57.971960 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:57.972849 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.973568 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.974818 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:57.975385 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:57.975502 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:57.979588 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:38505: BlockingRecv error: recv got EOF from 127.12.244.195:38505 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 02:20:58 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:58 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:58.464716 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:58.455285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:39355)
0926 02:20:58.456021 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:58.456040 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:58.456196 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:58.456611 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:58.456616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:58.457142 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:58.457302 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:58.457310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.458370 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.458374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:58.459349 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:58.459362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.460141 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.460146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:58.460270 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:58.460996 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:58.461014 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:58.464471 (+ 3457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":505,"threads_started":1}
W20260926 02:20:58.464923 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:58.455369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:39355 (local address 127.12.244.254:41137)
0926 02:20:58.455623 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:58.455633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:58.456140 (+ 507us) server_negotiation.cc:407] Connection header received
0926 02:20:58.456360 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:58.456368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:58.456456 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:58.456539 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:58.457442 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.458209 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.459509 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.459990 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.460118 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:58.464682 (+ 4564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:39355: BlockingRecv error: recv got EOF from 127.12.244.194:39355 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 02:20:58 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:58 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:58.941494 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:58.932605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:60445)
0926 02:20:58.933054 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:58.933075 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:58.933198 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:58.933578 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:58.933582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:58.933958 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:58.934101 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:58.934110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.935264 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.935269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:58.936246 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:58.936258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.937104 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.937130 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:58.937249 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:58.937936 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:58.937954 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:58.941149 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":266,"threads_started":1}
W20260926 02:20:58.941638 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:58.932676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:60445 (local address 127.12.244.254:41137)
0926 02:20:58.932937 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:58.932943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:58.933157 (+ 214us) server_negotiation.cc:407] Connection header received
0926 02:20:58.933304 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:58.933308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:58.933369 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:58.933454 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:58.934280 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.935078 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.936491 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.936948 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.937061 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:58.941363 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:60445: BlockingRecv error: recv got EOF from 127.12.244.193:60445 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 26 02:20:58 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:58 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:58.990468 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:58.980802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:56987)
0926 02:20:58.981393 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:58.981410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:58.981576 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:58.981998 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:58.982003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:58.982481 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:58.982663 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:58.982673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.983937 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.983946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:58.985008 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:58.985019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.985803 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.985811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:58.986010 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:58.986675 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:58.986695 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:58.990054 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":309,"threads_started":1}
W20260926 02:20:58.990665 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:58.980879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:56987 (local address 127.12.244.254:41137)
0926 02:20:58.981194 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:58.981203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:58.981497 (+ 294us) server_negotiation.cc:407] Connection header received
0926 02:20:58.981776 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:58.981782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:58.981855 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:58.981967 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:58.982823 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.983789 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.985167 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:58.985626 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:58.985765 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:58.990316 (+ 4551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:56987: BlockingRecv error: recv got EOF from 127.12.244.195:56987 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 26 02:20:59 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:59 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:59.475840 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:59.465865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:53809)
0926 02:20:59.466388 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:59.466405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:59.466517 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:59.466908 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:59.466911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:59.467333 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:59.467476 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:59.467485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.469188 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.469192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:59.470140 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:59.470152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.471468 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.471473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:59.471635 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:59.472311 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:59.472339 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:59.475586 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":299,"threads_started":1}
W20260926 02:20:59.476220 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:59.465952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:53809 (local address 127.12.244.254:41137)
0926 02:20:59.466176 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:59.466185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:59.466489 (+ 304us) server_negotiation.cc:407] Connection header received
0926 02:20:59.466638 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:59.466646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:59.466743 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:59.466877 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:59.467639 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.469024 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.470313 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.471292 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.471438 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:59.475922 (+ 4484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:53809: BlockingRecv error: recv got EOF from 127.12.244.194:53809 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 02:20:59 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:59 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:20:59.951597 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:59.942590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:39183)
0926 02:20:59.943189 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:59.943211 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:59.943346 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:59.943678 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:59.943682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:59.944120 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:59.944310 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:59.944319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.945497 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.945503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:59.946571 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:59.946586 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.947394 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.947400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:59.947575 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:59.948179 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:59.948197 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:20:59.951211 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":350,"threads_started":1}
W20260926 02:20:59.951785 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:59.942802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:39183 (local address 127.12.244.254:41137)
0926 02:20:59.943030 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:59.943034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:59.943294 (+ 260us) server_negotiation.cc:407] Connection header received
0926 02:20:59.943450 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:59.943455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:59.943525 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:59.943620 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:59.944489 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.945344 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.946724 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.947231 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.947370 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:59.951528 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:39183: BlockingRecv error: recv got EOF from 127.12.244.193:39183 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 26 02:20:59 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:20:59 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:00.000466 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:20:59.991890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:53543)
0926 02:20:59.992464 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 02:20:59.992481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:20:59.992622 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:20:59.992964 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:20:59.992967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:20:59.993330 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:20:59.993472 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:59.993480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.994551 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.994557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:59.995712 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:20:59.995724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.996515 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.996520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:20:59.996682 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:20:59.997262 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:20:59.997300 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:00.000233 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":263,"threads_started":1}
W20260926 02:21:00.000806 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:20:59.991955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:53543 (local address 127.12.244.254:41137)
0926 02:20:59.992200 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:20:59.992210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:20:59.992567 (+ 357us) server_negotiation.cc:407] Connection header received
0926 02:20:59.992736 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:20:59.992740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:20:59.992834 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:20:59.992920 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:20:59.993614 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.994401 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.995848 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:20:59.996367 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:20:59.996503 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:00.000442 (+ 3939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:53543: BlockingRecv error: recv got EOF from 127.12.244.195:53543 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 02:21:00 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:00 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:00.490397 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:00.476800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:46423)
0926 02:21:00.477403 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:00.477419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:00.477562 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:00.478006 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:00.478009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:00.478427 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:00.478602 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:00.478610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.480192 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.480198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:00.481452 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:00.481464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.482156 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.482160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:00.482287 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:00.486869 (+ 4582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:00.486895 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:00.490030 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":375,"threads_started":1}
W20260926 02:21:00.490515 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:00.477015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:46423 (local address 127.12.244.254:41137)
0926 02:21:00.477260 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:00.477266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:00.477514 (+ 248us) server_negotiation.cc:407] Connection header received
0926 02:21:00.477764 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:00.477769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:00.477860 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:00.477953 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:00.478754 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.479933 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.481601 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.482020 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.482127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:00.490260 (+ 8133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:46423: BlockingRecv error: recv got EOF from 127.12.244.194:46423 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 02:21:00 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:00 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:00.960940 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:00.952572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:32853)
0926 02:21:00.953117 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:00.953144 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:00.953288 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:00.953631 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:00.953637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:00.954073 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:00.954223 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:00.954231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.955309 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.955316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:00.956257 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:00.956269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.956978 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.956984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:00.957183 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:00.957852 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:00.957869 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:00.960695 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":295,"threads_started":1}
W20260926 02:21:00.961143 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:00.952854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:32853 (local address 127.12.244.254:41137)
0926 02:21:00.953091 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:00.953098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:00.953223 (+ 125us) server_negotiation.cc:407] Connection header received
0926 02:21:00.953422 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:00.953428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:00.953493 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:00.953594 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:00.954410 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.955176 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.956404 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:00.956821 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:00.956949 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:00.960915 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:32853: BlockingRecv error: recv got EOF from 127.12.244.193:32853 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 02:21:01 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:01 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:01.010983 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:01.001628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:40759)
0926 02:21:01.002154 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:01.002171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:01.002280 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:01.002549 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:01.002553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:01.002929 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:01.003169 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:01.003179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.004237 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.004242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:01.005290 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:01.005305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.006390 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.006397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:01.006626 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:01.007408 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:01.007428 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:01.010710 (+ 3282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":280,"threads_started":1}
W20260926 02:21:01.011224 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:01.001640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:40759 (local address 127.12.244.254:41137)
0926 02:21:01.001806 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:01.001811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:01.002242 (+ 431us) server_negotiation.cc:407] Connection header received
0926 02:21:01.002363 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:01.002367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:01.002428 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:01.002509 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:01.003328 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.004078 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.005502 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.006177 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.006286 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:01.010946 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:40759: BlockingRecv error: recv got EOF from 127.12.244.195:40759 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 02:21:01 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:01 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:01.499779 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:01.491316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:33577)
0926 02:21:01.491911 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:01.491934 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:01.492054 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:01.492363 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:01.492366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:01.492726 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:01.492873 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:01.492881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.493855 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.493859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:01.494799 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:01.494811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.495489 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.495493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:01.495674 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:01.496428 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:01.496450 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:01.499564 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":316,"threads_started":1}
W20260926 02:21:01.500008 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:01.491395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:33577 (local address 127.12.244.254:41137)
0926 02:21:01.491572 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:01.491576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:01.492012 (+ 436us) server_negotiation.cc:407] Connection header received
0926 02:21:01.492152 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:01.492156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:01.492217 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:01.492308 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:01.492994 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.493705 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.494923 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.495366 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.495455 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:01.499735 (+ 4280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:33577: BlockingRecv error: recv got EOF from 127.12.244.194:33577 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 02:21:01 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:01 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:01.971225 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:01.962142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:37255)
0926 02:21:01.962828 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:01.962847 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:01.963023 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:01.963402 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:01.963407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:01.963939 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:01.964119 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:01.964128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.965215 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.965222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:01.966237 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:01.966250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.967023 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.967029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:01.967242 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:01.967884 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:01.967901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:01.970916 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":351,"threads_started":1}
W20260926 02:21:01.971573 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:01.962356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:37255 (local address 127.12.244.254:41137)
0926 02:21:01.962641 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:01.962648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:01.962945 (+ 297us) server_negotiation.cc:407] Connection header received
0926 02:21:01.963193 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:01.963197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:01.963259 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:01.963377 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:01.964284 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.965056 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.966399 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:01.966812 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:01.966966 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:01.971269 (+ 4303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:37255: BlockingRecv error: recv got EOF from 127.12.244.193:37255 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 02:21:02 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:02 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:02.021559 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:02.012087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:56527)
0926 02:21:02.012828 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:02.012845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:02.013005 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:02.013376 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:02.013381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:02.013800 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:02.013955 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:02.013964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.015340 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.015346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:02.016440 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:02.016451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.017216 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.017223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:02.017414 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:02.018068 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:02.018086 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:02.021229 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":513,"threads_started":1}
W20260926 02:21:02.021718 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:02.012313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:56527 (local address 127.12.244.254:41137)
0926 02:21:02.012589 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:02.012596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:02.012932 (+ 336us) server_negotiation.cc:407] Connection header received
0926 02:21:02.013138 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:02.013142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:02.013233 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:02.013345 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:02.014105 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.015172 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.016609 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.017039 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.017180 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:02.021468 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:56527: BlockingRecv error: recv got EOF from 127.12.244.195:56527 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 02:21:02 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:02 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:02.509727 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:02.500698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:36363)
0926 02:21:02.501312 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:02.501327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:02.501482 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:02.501844 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:02.501849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:02.502273 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:02.502424 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:02.502432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.503515 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.503525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:02.504489 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:02.504504 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.505249 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.505254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:02.505391 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:02.506016 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:02.506034 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:02.509387 (+ 3353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":289,"threads_started":1}
W20260926 02:21:02.509821 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:02.500973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:36363 (local address 127.12.244.254:41137)
0926 02:21:02.501202 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:02.501209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:02.501410 (+ 201us) server_negotiation.cc:407] Connection header received
0926 02:21:02.501640 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:02.501645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:02.501705 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:02.501815 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:02.502570 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.503377 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.504646 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.505066 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.505204 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:02.509572 (+ 4368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:36363: BlockingRecv error: recv got EOF from 127.12.244.194:36363 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 02:21:02 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:02 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:02.980731 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:02.972141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:40907)
0926 02:21:02.972631 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:02.972652 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:02.972832 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:02.973153 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:02.973157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:02.973551 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:02.973720 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:02.973728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.974721 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.974727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:02.975688 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:02.975699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.976401 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.976407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:02.976577 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:02.977263 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:02.977282 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:02.980478 (+ 3196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":275,"threads_started":1}
W20260926 02:21:02.980957 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:02.972262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:40907 (local address 127.12.244.254:41137)
0926 02:21:02.972500 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:02.972507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:02.972733 (+ 226us) server_negotiation.cc:407] Connection header received
0926 02:21:02.972956 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:02.972966 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:02.973027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:02.973124 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:02.973869 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.974583 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.975832 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:02.976250 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:02.976388 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:02.980721 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:40907: BlockingRecv error: recv got EOF from 127.12.244.193:40907 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 02:21:03 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:03 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:03.032013 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:03.022639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:36137)
0926 02:21:03.023269 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:03.023292 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:03.023477 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:03.023802 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:03.023805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:03.024255 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:03.024413 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:03.024421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.025701 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.025708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:03.026740 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:03.026755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.027584 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.027589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:03.027734 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:03.028330 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:03.028349 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:03.031629 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":300,"threads_started":1}
W20260926 02:21:03.032104 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:03.022720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:36137 (local address 127.12.244.254:41137)
0926 02:21:03.023082 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:03.023114 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:03.023368 (+ 254us) server_negotiation.cc:407] Connection header received
0926 02:21:03.023615 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:03.023619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:03.023681 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:03.023764 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:03.024549 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.025559 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.026892 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.027413 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.027537 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:03.031857 (+ 4320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:36137: BlockingRecv error: recv got EOF from 127.12.244.195:36137 (error 108)
Metrics: {"server-negotiator.queue_time_us":206}
Sep 26 02:21:03 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:03 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:03.519176 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:03.510730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:41039)
0926 02:21:03.511272 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:03.511287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:03.511442 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:03.511787 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:03.511791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:03.512162 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:03.512345 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:03.512354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.513365 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.513371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:03.514289 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:03.514300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.515022 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.515027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:03.515247 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:03.515901 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:03.515921 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:03.518789 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
W20260926 02:21:03.519344 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:03.510821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:41039 (local address 127.12.244.254:41137)
0926 02:21:03.511062 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:03.511068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:03.511373 (+ 305us) server_negotiation.cc:407] Connection header received
0926 02:21:03.511565 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:03.511574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:03.511648 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:03.511753 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:03.512495 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.513222 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.514434 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.514875 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.515007 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:03.519069 (+ 4062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:41039: BlockingRecv error: recv got EOF from 127.12.244.194:41039 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 02:21:03 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:03 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.193@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:03.990159 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:03.981858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.193:38295)
0926 02:21:03.982507 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:03.982528 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:03.982638 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:03.982967 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:03.982970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:03.983417 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:03.983587 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:03.983596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.984599 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.984603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:03.985551 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:03.985563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.986204 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.986208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:03.986328 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:03.986976 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:03.986993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:03.989896 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":439,"threads_started":1}
W20260926 02:21:03.990330 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:03.981965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:38295 (local address 127.12.244.254:41137)
0926 02:21:03.982181 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:03.982189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:03.982602 (+ 413us) server_negotiation.cc:407] Connection header received
0926 02:21:03.982734 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:03.982737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:03.982835 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:03.982926 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:03.983724 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.984452 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.985681 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:03.986084 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:03.986178 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:03.990095 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.193:38295: BlockingRecv error: recv got EOF from 127.12.244.193:38295 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 02:21:04 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:04 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.195@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:04.041518 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:04.033040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.195:33559)
0926 02:21:04.033547 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:04.033563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:04.033719 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:04.034054 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:04.034059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:04.034435 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:04.034610 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:04.034619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.035748 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.035758 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:04.036681 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:04.036692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.037426 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.037430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:04.037555 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:04.038267 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:04.038292 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:04.041174 (+ 2882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":303,"threads_started":1}
W20260926 02:21:04.041806 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:04.033116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:33559 (local address 127.12.244.254:41137)
0926 02:21:04.033316 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:04.033323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:04.033647 (+ 324us) server_negotiation.cc:407] Connection header received
0926 02:21:04.033846 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:04.033850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:04.033919 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:04.034030 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:04.034757 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.035601 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.036848 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.037286 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.037385 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:04.041417 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.195:33559: BlockingRecv error: recv got EOF from 127.12.244.195:33559 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 02:21:04 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 02:21:04 dist-test-slave-qfmw krb5kdc[21570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.244.194@KRBTEST.COM for oryx/127.12.244.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 02:21:04.528443 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:04.520466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:41137 (local address 127.12.244.194:51279)
0926 02:21:04.520903 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:04.520918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:04.521024 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:04.521316 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:04.521319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:04.521707 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:04.521864 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:04.521873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.522829 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.522832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:04.523806 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:04.523817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.524575 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.524579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:04.524686 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:04.525378 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:04.525418 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:04.528152 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.244.254:41137: Server oryx/127.12.244.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":245,"threads_started":1}
W20260926 02:21:04.528604 22315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:04.520552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:51279 (local address 127.12.244.254:41137)
0926 02:21:04.520824 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:04.520830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:04.520990 (+ 160us) server_negotiation.cc:407] Connection header received
0926 02:21:04.521129 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:04.521133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:04.521192 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:04.521264 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:04.521988 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.522703 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.523933 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.524436 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.524532 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:04.528343 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.244.194:51279: BlockingRecv error: recv got EOF from 127.12.244.194:51279 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 02:21:04.866982 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22165
I20260926 02:21:04.880182 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:04.868357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.12.244.193:45109)
0926 02:21:04.868979 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:04.868987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:04.869017 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:21:04.869082 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:04.869089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:04.869336 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:04.869503 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:04.871144 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.872235 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.873587 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.874161 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.874271 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:04.876987 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:04.876996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:04.877011 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:04.877053 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:04.879643 (+ 2590us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.244.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:21:04.879941 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45340: Request ticket server kudu/127.12.244.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":321,"threads_started":1}
W20260926 02:21:04.880319 22627 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45340: Request ticket server kudu/127.12.244.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:21:04.880445 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22165: Not authorized: Client connection negotiation failed: client connection to 127.12.244.193:45109: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.244.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:21:04.880666 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22165
I20260926 02:21:04.897670 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22317
I20260926 02:21:04.910444 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:04.898822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.12.244.194:46191)
0926 02:21:04.899439 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:04.899445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:04.899477 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:21:04.899537 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:04.899543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:04.899793 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:04.899949 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:04.901178 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.902250 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.903567 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.904122 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.904235 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:04.907044 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:04.907054 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:04.907070 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:04.907196 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:04.909913 (+ 2717us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.244.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:21:04.910201 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37072: Request ticket server kudu/127.12.244.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":287,"threads_started":1}
W20260926 02:21:04.910557 22628 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37072: Request ticket server kudu/127.12.244.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:21:04.910614 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22317: Not authorized: Client connection negotiation failed: client connection to 127.12.244.194:46191: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.244.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:21:04.910815 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22317
I20260926 02:21:04.927383 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22448
I20260926 02:21:04.940560 22629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:04.928405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.12.244.195:34417)
0926 02:21:04.929047 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:04.929054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:04.929081 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:21:04.929143 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:04.929149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:04.929396 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:04.929557 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:04.931234 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.932369 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.933806 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:04.934384 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:04.934499 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:04.937279 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:04.937295 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:04.937331 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:04.937381 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:04.940044 (+ 2663us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.244.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 02:21:04.940333 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42386: Request ticket server kudu/127.12.244.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":309,"threads_started":1}
W20260926 02:21:04.940685 22629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42386: Request ticket server kudu/127.12.244.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 02:21:04.940819 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22448: Not authorized: Client connection negotiation failed: client connection to 127.12.244.195:34417: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.244.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 02:21:04.941004 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22448
I20260926 02:21:04.957731 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22095
I20260926 02:21:04.959019 22157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:05.075076 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22095
2026-09-26T02:21:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21597 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T02:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:21:05Z Disabled control of system clock
I20260926 02:21:05.154860 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:32823
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:32823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 02:21:05.510591 22634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:05.510908 22634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:05.510985 22634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:05.518558 22634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:21:05.518679 22634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:05.518718 22634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:05.518781 22634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:21:05.518824 22634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:21:05.528144 22634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:32823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:32823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:05.529564 22634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:05.531394 22634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:05.544248 22640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:05.544191 22642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:05.544224 22639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:05.544631 22634 server_base.cc:1034] running on GCE node
I20260926 02:21:05.545795 22634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:05.547398 22634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:05.548611 22634 hybrid_clock.cc:648] HybridClock initialized: now 1790389265548566 us; error 38 us; skew 500 ppm
I20260926 02:21:05.549048 22634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:05.559707 22634 webserver.cc:460] Webserver started at http://127.12.244.254:36043/ using document root <none> and password file <none>
I20260926 02:21:05.560527 22634 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:05.560631 22634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:05.561007 22634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:05.564199 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "620c5d4d71a94cf29bbe71c5b5a88e05"
format_stamp: "Formatted at 2026-09-26 02:21:05 on dist-test-slave-qfmw"
server_key: "a068545fe74c5b903662b609d073c2d2"
server_key_iv: "1d6af482654dffeaaafc2f5f8ca042a6"
server_key_version: "encryptionkey@0"
I20260926 02:21:05.565083 22634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "620c5d4d71a94cf29bbe71c5b5a88e05"
format_stamp: "Formatted at 2026-09-26 02:21:05 on dist-test-slave-qfmw"
server_key: "a068545fe74c5b903662b609d073c2d2"
server_key_iv: "1d6af482654dffeaaafc2f5f8ca042a6"
server_key_version: "encryptionkey@0"
I20260926 02:21:05.570719 22634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:21:05.574750 22648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:05.576488 22634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:21:05.576691 22634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "620c5d4d71a94cf29bbe71c5b5a88e05"
format_stamp: "Formatted at 2026-09-26 02:21:05 on dist-test-slave-qfmw"
server_key: "a068545fe74c5b903662b609d073c2d2"
server_key_iv: "1d6af482654dffeaaafc2f5f8ca042a6"
server_key_version: "encryptionkey@0"
I20260926 02:21:05.576980 22634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:05.606081 22634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:05.607034 22634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:05.607398 22634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:05.627498 22634 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:32823
I20260926 02:21:05.627521 22700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:32823 every 8 connection(s)
I20260926 02:21:05.628932 22634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:21:05.630874 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22634
I20260926 02:21:05.631114 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:21:05.632041 13267 external_mini_cluster.cc:1428] Setting key 8a427e75cd6671ba1c489c23fa59e8f8
I20260926 02:21:05.634418 22701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:21:05.646307 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:05.634395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.12.244.254:32823)
0926 02:21:05.635533 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:05.635544 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:05.635588 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:21:05.637222 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:05.637246 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:05.637771 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:05.638212 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:05.638843 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:05.640287 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:05.641644 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:05.642336 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:05.642535 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:05.643850 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:05.643859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:05.643874 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:05.644477 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:05.644560 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:05.644585 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:05.644726 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:05.644973 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 02:21:05.645321 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":408,"threads_started":1}
I20260926 02:21:05.647452 22701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05: Bootstrap starting.
I20260926 02:21:05.650979 22701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05: Neither blocks nor log segments found. Creating new log.
I20260926 02:21:05.652294 22701 log.cc:824] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05: Log is configured to *not* fsync() on all Append() calls
I20260926 02:21:05.655393 22701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05: No bootstrap required, opened a new log
I20260926 02:21:05.663856 22701 raft_consensus.cc:348] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32823 } }
I20260926 02:21:05.664319 22701 raft_consensus.cc:374] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:21:05.664441 22701 raft_consensus.cc:729] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 620c5d4d71a94cf29bbe71c5b5a88e05, State: Initialized, Role: FOLLOWER
I20260926 02:21:05.665095 22701 consensus_queue.cc:260] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32823 } }
I20260926 02:21:05.665367 22701 raft_consensus.cc:388] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:21:05.665526 22701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:21:05.665724 22701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:21:05.668540 22701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32823 } }
I20260926 02:21:05.669062 22701 leader_election.cc:302] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 620c5d4d71a94cf29bbe71c5b5a88e05; no voters:
I20260926 02:21:05.669644 22701 leader_election.cc:288] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:21:05.669826 22706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:21:05.670564 22706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [term 1 LEADER]: Becoming Leader. State: Replica: 620c5d4d71a94cf29bbe71c5b5a88e05, State: Running, Role: LEADER
I20260926 02:21:05.671255 22706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32823 } }
I20260926 02:21:05.672017 22701 sys_catalog.cc:564] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:21:05.673938 22708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 620c5d4d71a94cf29bbe71c5b5a88e05. Latest consensus state: current_term: 1 leader_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32823 } } }
I20260926 02:21:05.674324 22708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [sys.catalog]: This master's current role is: LEADER
I20260926 02:21:05.674993 22707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "620c5d4d71a94cf29bbe71c5b5a88e05" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32823 } } }
I20260926 02:21:05.675438 22707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05 [sys.catalog]: This master's current role is: LEADER
I20260926 02:21:05.677326 22711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:21:05.683516 22711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:21:05.694039 22711 catalog_manager.cc:1347] Generated new cluster ID: 0e35f5e0e8c34072942c0375cd00fddc
I20260926 02:21:05.694170 22711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:21:05.720093 22711 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:21:05.721549 22711 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:21:05.730074 22711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 620c5d4d71a94cf29bbe71c5b5a88e05: Generated new TSK 0
I20260926 02:21:05.731140 22711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:21:05.748457 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:32823
--builtin_ntp_servers=127.12.244.212:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 02:21:06.117189 22725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:06.117504 22725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:06.117643 22725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:06.125823 22725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:06.125969 22725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:06.126163 22725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:21:06.135800 22725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:32823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:06.137253 22725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:06.139223 22725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:06.154053 22731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:06.154386 22733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:06.154633 22730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:06.155210 22725 server_base.cc:1034] running on GCE node
I20260926 02:21:06.155905 22725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:06.157017 22725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:06.158264 22725 hybrid_clock.cc:648] HybridClock initialized: now 1790389266158209 us; error 49 us; skew 500 ppm
I20260926 02:21:06.158645 22725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:06.161193 22725 webserver.cc:460] Webserver started at http://127.12.244.193:42107/ using document root <none> and password file <none>
I20260926 02:21:06.161968 22725 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:06.162103 22725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:06.162436 22725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:06.165549 22725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "34e28cedfd4744e2a81cb3273fe4c540"
format_stamp: "Formatted at 2026-09-26 02:21:06 on dist-test-slave-qfmw"
server_key: "b39240bf61bf65c4dfdce503b6b7cb4b"
server_key_iv: "5392ff1f601bfb5c12ced4286b10521b"
server_key_version: "encryptionkey@0"
I20260926 02:21:06.166427 22725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "34e28cedfd4744e2a81cb3273fe4c540"
format_stamp: "Formatted at 2026-09-26 02:21:06 on dist-test-slave-qfmw"
server_key: "b39240bf61bf65c4dfdce503b6b7cb4b"
server_key_iv: "5392ff1f601bfb5c12ced4286b10521b"
server_key_version: "encryptionkey@0"
I20260926 02:21:06.172328 22725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:21:06.176327 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:06.177949 22725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:21:06.178155 22725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "34e28cedfd4744e2a81cb3273fe4c540"
format_stamp: "Formatted at 2026-09-26 02:21:06 on dist-test-slave-qfmw"
server_key: "b39240bf61bf65c4dfdce503b6b7cb4b"
server_key_iv: "5392ff1f601bfb5c12ced4286b10521b"
server_key_version: "encryptionkey@0"
I20260926 02:21:06.178403 22725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:06.201426 22725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:06.202375 22725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:06.202703 22725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:06.203735 22725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:06.205348 22725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:06.205471 22725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:06.205600 22725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:06.205687 22725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:06.239657 22725 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:35905
I20260926 02:21:06.239758 22852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:35905 every 8 connection(s)
I20260926 02:21:06.241014 22725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:21:06.246902 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22725
I20260926 02:21:06.247094 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:21:06.247766 13267 external_mini_cluster.cc:1428] Setting key 99b86a954b954feef5f6cf299c9de161
I20260926 02:21:06.250537 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:32823
--builtin_ntp_servers=127.12.244.212:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:21:06.254781 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:06.243747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:38145 (local address 127.12.244.254:32823)
0926 02:21:06.244179 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:06.244184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:06.245537 (+ 1353us) server_negotiation.cc:407] Connection header received
0926 02:21:06.247645 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:06.247651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:06.247756 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:06.247875 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:06.249174 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.250030 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.251813 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.252211 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.252397 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:06.253867 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:06.253878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:06.253882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:06.253902 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:06.253942 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:06.253955 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:06.254052 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:06.254325 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:21:06.254429 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260926 02:21:06.255964 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:06.244384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:32823 (local address 127.12.244.193:38145)
0926 02:21:06.245354 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:06.245397 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:06.247331 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:06.248127 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:06.248136 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:06.248239 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:06.248992 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:06.249010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.250199 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.250203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:06.251617 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:06.251630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.252477 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.252484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:06.252736 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:06.253489 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:21:06.253491 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:06.253534 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:21:06.253625 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:06.254051 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:06.254063 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:06.254073 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:21:06.254274 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 02:21:06.254727 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":274,"threads_started":1}
I20260926 02:21:06.257701 22853 heartbeater.cc:347] Connected to a master server at 127.12.244.254:32823
I20260926 02:21:06.258033 22853 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:06.258826 22853 heartbeater.cc:511] Master 127.12.244.254:32823 requested a full tablet report, sending...
I20260926 02:21:06.261253 22666 ts_manager.cc:194] Registered new tserver with Master: 34e28cedfd4744e2a81cb3273fe4c540 (127.12.244.193:35905)
I20260926 02:21:06.263523 22666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.193:38145
W20260926 02:21:06.594453 22857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:06.594756 22857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:06.594846 22857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:06.602238 22857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:06.602373 22857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:06.602481 22857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:21:06.612082 22857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:32823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:06.613397 22857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:06.615305 22857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:06.629462 22863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:06.629570 22862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:06.630010 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:06.630717 22857 server_base.cc:1034] running on GCE node
I20260926 02:21:06.631453 22857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:06.632498 22857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:06.633733 22857 hybrid_clock.cc:648] HybridClock initialized: now 1790389266633659 us; error 74 us; skew 500 ppm
I20260926 02:21:06.634083 22857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:06.636725 22857 webserver.cc:460] Webserver started at http://127.12.244.194:42411/ using document root <none> and password file <none>
I20260926 02:21:06.637521 22857 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:06.637648 22857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:06.637979 22857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:06.640930 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "59ba1a60d61e465eb43de50a3ad331b9"
format_stamp: "Formatted at 2026-09-26 02:21:06 on dist-test-slave-qfmw"
server_key: "256726a6bfdf712c1ed740d0f8aa8d4a"
server_key_iv: "16971c64272528881ffa57771ae70718"
server_key_version: "encryptionkey@0"
I20260926 02:21:06.641743 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "59ba1a60d61e465eb43de50a3ad331b9"
format_stamp: "Formatted at 2026-09-26 02:21:06 on dist-test-slave-qfmw"
server_key: "256726a6bfdf712c1ed740d0f8aa8d4a"
server_key_iv: "16971c64272528881ffa57771ae70718"
server_key_version: "encryptionkey@0"
I20260926 02:21:06.647248 22857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:21:06.650928 22871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:06.652495 22857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:21:06.652796 22857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "59ba1a60d61e465eb43de50a3ad331b9"
format_stamp: "Formatted at 2026-09-26 02:21:06 on dist-test-slave-qfmw"
server_key: "256726a6bfdf712c1ed740d0f8aa8d4a"
server_key_iv: "16971c64272528881ffa57771ae70718"
server_key_version: "encryptionkey@0"
I20260926 02:21:06.653108 22857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:06.693205 22857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:06.694252 22857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:06.694597 22857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:06.696112 22857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:06.698995 22857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:06.699144 22857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:06.699316 22857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:06.699411 22857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:06.734474 22857 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:46417
I20260926 02:21:06.734565 22985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:46417 every 8 connection(s)
I20260926 02:21:06.735925 22857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:21:06.736497 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22857
I20260926 02:21:06.736682 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:21:06.737257 13267 external_mini_cluster.cc:1428] Setting key 0f4d0c8c95f55b0634fd6afad280a760
I20260926 02:21:06.740823 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:32823
--builtin_ntp_servers=127.12.244.212:35853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 02:21:06.752585 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:06.739071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:47621 (local address 127.12.244.254:32823)
0926 02:21:06.739273 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:06.739279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:06.741267 (+ 1988us) server_negotiation.cc:407] Connection header received
0926 02:21:06.744220 (+ 2953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:06.744225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:06.744305 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:06.744407 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:06.745953 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.746723 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.749208 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.749933 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.750077 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:06.751571 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:06.751580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:06.751587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:06.751615 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:06.751658 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:06.751676 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:06.751770 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:06.752037 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:21:06.752165 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 02:21:06.753530 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:06.739914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:32823 (local address 127.12.244.194:47621)
0926 02:21:06.741100 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:06.741149 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:06.743854 (+ 2705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:06.744706 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:06.744722 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:06.744866 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:06.745740 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:06.745779 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.746890 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.746898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:06.749008 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:06.749020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:06.750154 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:06.750158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:06.750386 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:06.751217 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:21:06.751219 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:06.751265 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:21:06.751357 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:06.751815 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:06.751822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:06.751828 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:21:06.751977 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 02:21:06.752320 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":295,"threads_started":1}
I20260926 02:21:06.755131 22986 heartbeater.cc:347] Connected to a master server at 127.12.244.254:32823
I20260926 02:21:06.755455 22986 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:06.756222 22986 heartbeater.cc:511] Master 127.12.244.254:32823 requested a full tablet report, sending...
I20260926 02:21:06.757856 22666 ts_manager.cc:194] Registered new tserver with Master: 59ba1a60d61e465eb43de50a3ad331b9 (127.12.244.194:46417)
I20260926 02:21:06.759181 22666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.194:47621
W20260926 02:21:07.093621 22989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:07.093971 22989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:07.094058 22989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:07.101625 22989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:07.101758 22989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:07.101866 22989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:21:07.111245 22989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:32823
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:07.112578 22989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:07.114390 22989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:07.128677 22994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:07.128703 22997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:07.129065 22995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:07.129747 22989 server_base.cc:1034] running on GCE node
I20260926 02:21:07.130394 22989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:07.131520 22989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:07.132789 22989 hybrid_clock.cc:648] HybridClock initialized: now 1790389267132726 us; error 47 us; skew 500 ppm
I20260926 02:21:07.133213 22989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:07.135977 22989 webserver.cc:460] Webserver started at http://127.12.244.195:39187/ using document root <none> and password file <none>
I20260926 02:21:07.136771 22989 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:07.136900 22989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:07.137270 22989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:07.140457 22989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "ab64bdeaee94403eb2c1b34b0b7cac2f"
format_stamp: "Formatted at 2026-09-26 02:21:07 on dist-test-slave-qfmw"
server_key: "febb3883aa07ffcf8f91bbc24973a261"
server_key_iv: "8f940c4da2ddfe0cf5512a5caf199bf5"
server_key_version: "encryptionkey@0"
I20260926 02:21:07.141350 22989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "ab64bdeaee94403eb2c1b34b0b7cac2f"
format_stamp: "Formatted at 2026-09-26 02:21:07 on dist-test-slave-qfmw"
server_key: "febb3883aa07ffcf8f91bbc24973a261"
server_key_iv: "8f940c4da2ddfe0cf5512a5caf199bf5"
server_key_version: "encryptionkey@0"
I20260926 02:21:07.147202 22989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:21:07.151202 23003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:07.152871 22989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:21:07.153086 22989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "ab64bdeaee94403eb2c1b34b0b7cac2f"
format_stamp: "Formatted at 2026-09-26 02:21:07 on dist-test-slave-qfmw"
server_key: "febb3883aa07ffcf8f91bbc24973a261"
server_key_iv: "8f940c4da2ddfe0cf5512a5caf199bf5"
server_key_version: "encryptionkey@0"
I20260926 02:21:07.153340 22989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:07.170971 22989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:07.172025 22989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:07.172385 22989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:07.173480 22989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:07.175132 22989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:07.175256 22989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:07.175393 22989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:07.175477 22989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:07.214897 22989 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:34727
I20260926 02:21:07.214942 23115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:34727 every 8 connection(s)
I20260926 02:21:07.216440 22989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:21:07.219198 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 22989
I20260926 02:21:07.219384 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:21:07.219866 13267 external_mini_cluster.cc:1428] Setting key d49112a9802dd5e5a5bb91e86359884b
I20260926 02:21:07.230504 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:07.219391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:37423 (local address 127.12.244.254:32823)
0926 02:21:07.219629 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:07.219635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:07.221460 (+ 1825us) server_negotiation.cc:407] Connection header received
0926 02:21:07.223577 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:07.223581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:07.223837 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:07.223932 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:07.225119 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.225910 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.227774 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.228219 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.228341 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:07.229788 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:07.229794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:07.229798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:07.229819 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:07.229860 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:07.229873 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:07.229948 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:07.230165 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 02:21:07.230251 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:21:07.232048 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:07.220136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:32823 (local address 127.12.244.195:37423)
0926 02:21:07.221263 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:07.221304 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:07.223311 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:07.224165 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:07.224175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:07.224289 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:07.224915 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:07.224933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.226111 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.226116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:07.227603 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:07.227620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.228462 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.228467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:07.228691 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:07.229434 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:21:07.229436 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:07.229480 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:21:07.229574 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:07.229963 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:07.229971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:07.229977 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:21:07.230137 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 02:21:07.230507 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":387,"threads_started":1}
I20260926 02:21:07.233777 23116 heartbeater.cc:347] Connected to a master server at 127.12.244.254:32823
I20260926 02:21:07.234117 23116 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:07.234959 23116 heartbeater.cc:511] Master 127.12.244.254:32823 requested a full tablet report, sending...
I20260926 02:21:07.236436 22666 ts_manager.cc:194] Registered new tserver with Master: ab64bdeaee94403eb2c1b34b0b7cac2f (127.12.244.195:34727)
I20260926 02:21:07.237284 22666 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.195:37423
I20260926 02:21:07.248596 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:21:07.255344 22855 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:07.253474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.12.244.254:32823)
0926 02:21:07.253774 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:07.253781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:07.254100 (+ 319us) server_negotiation.cc:407] Connection header received
0926 02:21:07.254358 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:07.254362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:07.254429 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:07.254520 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:07.255052 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33734: BlockingRecv error: recv got EOF from 127.0.0.1:33734 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:21:07.256397 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22725
I20260926 02:21:07.265328 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:07.257401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.12.244.193:35905)
0926 02:21:07.258044 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:07.258052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:07.258078 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:21:07.258144 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:07.258151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:07.258435 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:07.258619 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:07.259140 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.260199 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.261694 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.262321 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.262473 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:07.263780 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:07.263790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:07.263805 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:07.264429 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:07.264486 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:07.264507 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:07.264652 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:07.264807 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 02:21:07.264961 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":283,"threads_started":1}
I20260926 02:21:07.266989 22848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:07.267302 22853 heartbeater.cc:503] Master 127.12.244.254:32823 was elected leader, sending a full tablet report...
I20260926 02:21:07.356014 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22725
I20260926 02:21:07.374444 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22857
I20260926 02:21:07.384078 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:07.375514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.12.244.194:46417)
0926 02:21:07.376005 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:07.376012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:07.376040 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:21:07.376100 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:07.376106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:07.376354 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:07.376491 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:07.377023 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.378413 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.380180 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.380902 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.381014 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:07.382285 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:07.382294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:07.382311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:07.382970 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:07.383024 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:07.383047 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:07.383236 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:07.383484 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 02:21:07.383666 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260926 02:21:07.385555 22981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:07.471879 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22857
I20260926 02:21:07.488550 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22989
I20260926 02:21:07.497800 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:07.489470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.12.244.195:34727)
0926 02:21:07.490101 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:07.490109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:07.490141 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:21:07.490208 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:07.490215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:07.490446 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:07.490624 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:07.491266 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.492407 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.493837 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:07.494476 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:07.494637 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:07.496150 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:07.496161 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:07.496178 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:07.496846 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:07.496900 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:07.496923 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:07.497081 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:07.497251 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 02:21:07.497399 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":320,"threads_started":1}
I20260926 02:21:07.499363 23111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:07.586980 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22989
I20260926 02:21:07.604753 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 22634
I20260926 02:21:07.606359 22696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:07.717082 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 22634
2026-09-26T02:21:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2639 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T02:21:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:21:07Z Disabled control of system clock
I20260926 02:21:07.798377 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:44593
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:44593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:21:08.153312 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:08.153642 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:08.153771 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:08.161258 23135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:21:08.161375 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:08.161412 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:08.161469 23135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:21:08.161512 23135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:21:08.170993 23135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:44593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:44593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:08.172448 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:08.174340 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:08.187398 23140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:08.187599 23141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:08.187347 23143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:08.188905 23135 server_base.cc:1034] running on GCE node
I20260926 02:21:08.189898 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:08.191601 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:08.192900 23135 hybrid_clock.cc:648] HybridClock initialized: now 1790389268192849 us; error 89 us; skew 500 ppm
I20260926 02:21:08.193325 23135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:08.196059 23135 webserver.cc:460] Webserver started at http://127.12.244.254:39601/ using document root <none> and password file <none>
I20260926 02:21:08.196895 23135 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:08.197023 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:08.197395 23135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:08.200701 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "3b984ea9164b4d0e98efa0d2400a830d"
format_stamp: "Formatted at 2026-09-26 02:21:08 on dist-test-slave-qfmw"
server_key: "7c45717b862ea698b22bca9ec07c08b2"
server_key_iv: "e7763b394e85301c79d8b799ec9dd44e"
server_key_version: "encryptionkey@0"
I20260926 02:21:08.201610 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "3b984ea9164b4d0e98efa0d2400a830d"
format_stamp: "Formatted at 2026-09-26 02:21:08 on dist-test-slave-qfmw"
server_key: "7c45717b862ea698b22bca9ec07c08b2"
server_key_iv: "e7763b394e85301c79d8b799ec9dd44e"
server_key_version: "encryptionkey@0"
I20260926 02:21:08.207358 23135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 02:21:08.211331 23149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:08.213017 23135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:21:08.213188 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "3b984ea9164b4d0e98efa0d2400a830d"
format_stamp: "Formatted at 2026-09-26 02:21:08 on dist-test-slave-qfmw"
server_key: "7c45717b862ea698b22bca9ec07c08b2"
server_key_iv: "e7763b394e85301c79d8b799ec9dd44e"
server_key_version: "encryptionkey@0"
I20260926 02:21:08.213393 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:08.219813 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:08.220651 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:08.220937 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:08.240012 23135 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:44593
I20260926 02:21:08.240041 23200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:44593 every 8 connection(s)
I20260926 02:21:08.241462 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:21:08.242972 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 23135
I20260926 02:21:08.243237 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:21:08.244067 13267 external_mini_cluster.cc:1428] Setting key 566f5b51ac048cb29801e0b4ea562298
I20260926 02:21:08.246433 23201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:21:08.252859 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:08.246236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.12.244.254:44593)
0926 02:21:08.247347 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:08.247358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:08.247402 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:21:08.248977 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:08.249001 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:08.249478 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:08.249926 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:08.251024 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:08.251031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:08.251044 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:08.251061 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:08.251197 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:08.251220 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:08.251478 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:08.251629 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:21:08.251892 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":359,"threads_started":1}
I20260926 02:21:08.259230 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d: Bootstrap starting.
I20260926 02:21:08.262621 23201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d: Neither blocks nor log segments found. Creating new log.
I20260926 02:21:08.263800 23201 log.cc:824] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:21:08.267215 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d: No bootstrap required, opened a new log
I20260926 02:21:08.275208 23201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b984ea9164b4d0e98efa0d2400a830d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44593 } }
I20260926 02:21:08.275614 23201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:21:08.275688 23201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b984ea9164b4d0e98efa0d2400a830d, State: Initialized, Role: FOLLOWER
I20260926 02:21:08.276315 23201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b984ea9164b4d0e98efa0d2400a830d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44593 } }
I20260926 02:21:08.276533 23201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:21:08.276635 23201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:21:08.276772 23201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:21:08.278949 23201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b984ea9164b4d0e98efa0d2400a830d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44593 } }
I20260926 02:21:08.279529 23201 leader_election.cc:302] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b984ea9164b4d0e98efa0d2400a830d; no voters:
I20260926 02:21:08.280279 23201 leader_election.cc:288] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:21:08.280397 23206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:21:08.281242 23206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [term 1 LEADER]: Becoming Leader. State: Replica: 3b984ea9164b4d0e98efa0d2400a830d, State: Running, Role: LEADER
I20260926 02:21:08.282019 23206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b984ea9164b4d0e98efa0d2400a830d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44593 } }
I20260926 02:21:08.282790 23201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:21:08.285288 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b984ea9164b4d0e98efa0d2400a830d. Latest consensus state: current_term: 1 leader_uuid: "3b984ea9164b4d0e98efa0d2400a830d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b984ea9164b4d0e98efa0d2400a830d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44593 } } }
I20260926 02:21:08.285418 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b984ea9164b4d0e98efa0d2400a830d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b984ea9164b4d0e98efa0d2400a830d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44593 } } }
I20260926 02:21:08.285739 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [sys.catalog]: This master's current role is: LEADER
I20260926 02:21:08.285737 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d [sys.catalog]: This master's current role is: LEADER
I20260926 02:21:08.291729 23215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:21:08.298110 23215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:21:08.308159 23215 catalog_manager.cc:1347] Generated new cluster ID: 6455ca944eb54d149f6c3acb94e858b6
I20260926 02:21:08.308306 23215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:21:08.333269 23215 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:21:08.333890 23215 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:21:08.344385 23215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b984ea9164b4d0e98efa0d2400a830d: Generated new TSK 0
I20260926 02:21:08.345384 23215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:21:08.350883 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:44593
--builtin_ntp_servers=127.12.244.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 02:21:08.731873 23225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:08.732187 23225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:08.732275 23225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:08.739812 23225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:08.739929 23225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:08.740044 23225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:21:08.749323 23225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:44593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:08.750623 23225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:08.752415 23225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:08.765915 23230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:08.765866 23233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:08.766178 23231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:08.767355 23225 server_base.cc:1034] running on GCE node
I20260926 02:21:08.768080 23225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:08.769146 23225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:08.770371 23225 hybrid_clock.cc:648] HybridClock initialized: now 1790389268770312 us; error 55 us; skew 500 ppm
I20260926 02:21:08.770685 23225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:08.773097 23225 webserver.cc:460] Webserver started at http://127.12.244.193:41087/ using document root <none> and password file <none>
I20260926 02:21:08.773804 23225 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:08.773895 23225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:08.774155 23225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:08.777200 23225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "a869b85f7e424c179da32994477b119c"
format_stamp: "Formatted at 2026-09-26 02:21:08 on dist-test-slave-qfmw"
server_key: "88e6a11872d9ec379e9f275827f54bd6"
server_key_iv: "a5ddac441459c42f3fc1e43c73652ec2"
server_key_version: "encryptionkey@0"
I20260926 02:21:08.778059 23225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "a869b85f7e424c179da32994477b119c"
format_stamp: "Formatted at 2026-09-26 02:21:08 on dist-test-slave-qfmw"
server_key: "88e6a11872d9ec379e9f275827f54bd6"
server_key_iv: "a5ddac441459c42f3fc1e43c73652ec2"
server_key_version: "encryptionkey@0"
I20260926 02:21:08.784027 23225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 02:21:08.788846 23239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:08.790520 23225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:21:08.790753 23225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "a869b85f7e424c179da32994477b119c"
format_stamp: "Formatted at 2026-09-26 02:21:08 on dist-test-slave-qfmw"
server_key: "88e6a11872d9ec379e9f275827f54bd6"
server_key_iv: "a5ddac441459c42f3fc1e43c73652ec2"
server_key_version: "encryptionkey@0"
I20260926 02:21:08.791013 23225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:08.797145 23225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:08.797958 23225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:08.798267 23225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:08.799305 23225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:08.800794 23225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:08.800881 23225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:08.800964 23225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:08.801018 23225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:08.834649 23225 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:37985
I20260926 02:21:08.834698 23352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:37985 every 8 connection(s)
I20260926 02:21:08.836109 23225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:21:08.839373 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 23225
I20260926 02:21:08.839591 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:21:08.840217 13267 external_mini_cluster.cc:1428] Setting key a2cc8b3258f3c61db4b50d720ddf61fc
I20260926 02:21:08.843120 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:44593
--builtin_ntp_servers=127.12.244.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 02:21:08.845477 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:08.838849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:35005 (local address 127.12.244.254:44593)
0926 02:21:08.839374 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:08.839381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:08.840849 (+ 1468us) server_negotiation.cc:407] Connection header received
0926 02:21:08.842775 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:08.842782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:08.842864 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:08.842993 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:08.844649 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:08.844659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:08.844664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:08.844692 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:08.844732 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:08.844744 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:08.844921 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:08.845142 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:21:08.845211 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":276,"threads_started":1}
I20260926 02:21:08.846526 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:08.839478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:44593 (local address 127.12.244.193:35005)
0926 02:21:08.840646 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:08.840691 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:08.842467 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:08.843348 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:08.843362 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:08.843523 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:08.844319 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:21:08.844322 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:08.844363 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:21:08.844455 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:08.844881 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:08.844893 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:08.844903 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:21:08.845236 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 02:21:08.845537 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260926 02:21:08.848363 23353 heartbeater.cc:347] Connected to a master server at 127.12.244.254:44593
I20260926 02:21:08.848726 23353 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:08.849437 23353 heartbeater.cc:511] Master 127.12.244.254:44593 requested a full tablet report, sending...
I20260926 02:21:08.851819 23166 ts_manager.cc:194] Registered new tserver with Master: a869b85f7e424c179da32994477b119c (127.12.244.193:37985)
W20260926 02:21:09.205929 23357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:09.206349 23357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:09.206512 23357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:09.214062 23357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:09.214257 23357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:09.214448 23357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:21:09.224565 23357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:44593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:09.226236 23357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:09.228361 23357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:09.244073 23363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:09.244150 23365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:09.244978 23362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:09.249137 23357 server_base.cc:1034] running on GCE node
I20260926 02:21:09.249884 23357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:09.251051 23357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:09.252483 23357 hybrid_clock.cc:648] HybridClock initialized: now 1790389269252428 us; error 42 us; skew 500 ppm
I20260926 02:21:09.252892 23357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:09.255878 23357 webserver.cc:460] Webserver started at http://127.12.244.194:45827/ using document root <none> and password file <none>
I20260926 02:21:09.256700 23357 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:09.256824 23357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:09.257210 23357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:09.260411 23357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "af31ebec92a34a67b1e94e2ca5a8ee56"
format_stamp: "Formatted at 2026-09-26 02:21:09 on dist-test-slave-qfmw"
server_key: "2092c4b0d7b1967f34b9573350183a3d"
server_key_iv: "994ddcc30324cd26edc3156b74b08eb8"
server_key_version: "encryptionkey@0"
I20260926 02:21:09.261291 23357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "af31ebec92a34a67b1e94e2ca5a8ee56"
format_stamp: "Formatted at 2026-09-26 02:21:09 on dist-test-slave-qfmw"
server_key: "2092c4b0d7b1967f34b9573350183a3d"
server_key_iv: "994ddcc30324cd26edc3156b74b08eb8"
server_key_version: "encryptionkey@0"
I20260926 02:21:09.267625 23357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260926 02:21:09.271896 23371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:09.273573 23357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:21:09.273793 23357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "af31ebec92a34a67b1e94e2ca5a8ee56"
format_stamp: "Formatted at 2026-09-26 02:21:09 on dist-test-slave-qfmw"
server_key: "2092c4b0d7b1967f34b9573350183a3d"
server_key_iv: "994ddcc30324cd26edc3156b74b08eb8"
server_key_version: "encryptionkey@0"
I20260926 02:21:09.274046 23357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:09.280452 23357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:09.281435 23357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:09.281755 23357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:09.282997 23357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:09.284863 23357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:09.284992 23357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:09.285144 23357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:09.285233 23357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:09.321275 23357 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:44991
I20260926 02:21:09.321345 23485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:44991 every 8 connection(s)
I20260926 02:21:09.322681 23357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:21:09.328788 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 23357
I20260926 02:21:09.328974 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:21:09.329521 13267 external_mini_cluster.cc:1428] Setting key 0ab8ee9afd9bbc551e937d197a321017
I20260926 02:21:09.332479 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:09.325665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:40637 (local address 127.12.244.254:44593)
0926 02:21:09.325922 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:09.325929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:09.327670 (+ 1741us) server_negotiation.cc:407] Connection header received
0926 02:21:09.329896 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:09.329901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:09.329973 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:09.330070 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:09.331734 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:09.331740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:09.331744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:09.331765 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:09.331810 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:09.331823 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:09.331937 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:09.332204 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:21:09.332258 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 02:21:09.332875 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:44593
--builtin_ntp_servers=127.12.244.212:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 02:21:09.333909 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:09.326319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:44593 (local address 127.12.244.194:40637)
0926 02:21:09.327480 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:09.327528 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:09.329523 (+ 1995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:09.330345 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:09.330380 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:09.330519 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:09.331287 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:21:09.331290 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:09.331396 (+ 106us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:21:09.331524 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:09.331932 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:09.331943 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:09.331951 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:21:09.332243 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 02:21:09.332544 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":382,"threads_started":1}
I20260926 02:21:09.335517 23486 heartbeater.cc:347] Connected to a master server at 127.12.244.254:44593
I20260926 02:21:09.335862 23486 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:09.336584 23486 heartbeater.cc:511] Master 127.12.244.254:44593 requested a full tablet report, sending...
I20260926 02:21:09.338280 23166 ts_manager.cc:194] Registered new tserver with Master: af31ebec92a34a67b1e94e2ca5a8ee56 (127.12.244.194:44991)
W20260926 02:21:09.678748 23489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:09.679162 23489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:09.679332 23489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:09.687004 23489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:09.687192 23489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:09.687361 23489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:21:09.697077 23489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:44593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:09.698482 23489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:09.700372 23489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:09.713973 23495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:09.714030 23494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:09.714054 23497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:09.714923 23489 server_base.cc:1034] running on GCE node
I20260926 02:21:09.715644 23489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:09.716692 23489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:09.717969 23489 hybrid_clock.cc:648] HybridClock initialized: now 1790389269717916 us; error 42 us; skew 500 ppm
I20260926 02:21:09.718389 23489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:21:09.720983 23489 webserver.cc:460] Webserver started at http://127.12.244.195:41579/ using document root <none> and password file <none>
I20260926 02:21:09.721782 23489 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:09.721902 23489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:09.722270 23489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:09.725315 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "d9eb2a80ff42439ca6b1251506621f33"
format_stamp: "Formatted at 2026-09-26 02:21:09 on dist-test-slave-qfmw"
server_key: "d32a49e62336e2be9cd8a1fd5bbfd738"
server_key_iv: "d3bfeeeb628eabbd6c4c534e2fb2f4f0"
server_key_version: "encryptionkey@0"
I20260926 02:21:09.726253 23489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "d9eb2a80ff42439ca6b1251506621f33"
format_stamp: "Formatted at 2026-09-26 02:21:09 on dist-test-slave-qfmw"
server_key: "d32a49e62336e2be9cd8a1fd5bbfd738"
server_key_iv: "d3bfeeeb628eabbd6c4c534e2fb2f4f0"
server_key_version: "encryptionkey@0"
I20260926 02:21:09.731956 23489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:21:09.735822 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:09.737358 23489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:21:09.737553 23489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "d9eb2a80ff42439ca6b1251506621f33"
format_stamp: "Formatted at 2026-09-26 02:21:09 on dist-test-slave-qfmw"
server_key: "d32a49e62336e2be9cd8a1fd5bbfd738"
server_key_iv: "d3bfeeeb628eabbd6c4c534e2fb2f4f0"
server_key_version: "encryptionkey@0"
I20260926 02:21:09.737883 23489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:09.743831 23489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:09.744666 23489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:09.744963 23489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:09.745900 23489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:09.747488 23489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:09.747601 23489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:09.747732 23489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:09.747838 23489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:09.781234 23489 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:45401
I20260926 02:21:09.781317 23615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:45401 every 8 connection(s)
I20260926 02:21:09.782614 23489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:21:09.789034 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 23489
I20260926 02:21:09.789227 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:21:09.789810 13267 external_mini_cluster.cc:1428] Setting key f90063cc091cc894b6f28bd77195fd12
I20260926 02:21:09.792951 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:09.785485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:51635 (local address 127.12.244.254:44593)
0926 02:21:09.785685 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:09.785690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:09.787375 (+ 1685us) server_negotiation.cc:407] Connection header received
0926 02:21:09.789897 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:09.789903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:09.789982 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:09.790084 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:09.792181 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:09.792185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:09.792188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:09.792205 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:09.792273 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:09.792287 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:09.792390 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:09.792617 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 02:21:09.792663 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:21:09.794050 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:09.786001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:44593 (local address 127.12.244.195:51635)
0926 02:21:09.787152 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:09.787208 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:09.789545 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:09.790496 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:09.790509 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:09.790651 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:09.791871 (+ 1220us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:21:09.791873 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:09.791916 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:21:09.792010 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:09.792405 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:09.792416 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:09.792424 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:21:09.792722 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 02:21:09.793089 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":277,"threads_started":1}
I20260926 02:21:09.795588 23617 heartbeater.cc:347] Connected to a master server at 127.12.244.254:44593
I20260926 02:21:09.795923 23617 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:09.796547 23617 heartbeater.cc:511] Master 127.12.244.254:44593 requested a full tablet report, sending...
I20260926 02:21:09.797922 23166 ts_manager.cc:194] Registered new tserver with Master: d9eb2a80ff42439ca6b1251506621f33 (127.12.244.195:45401)
I20260926 02:21:09.805812 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:21:09.812155 23355 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:21:09.810544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.12.244.254:44593)
0926 02:21:09.810711 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:09.810717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:09.811271 (+ 554us) server_negotiation.cc:407] Connection header received
0926 02:21:09.811421 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:09.811425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:09.811476 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:09.811564 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:09.811848 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60168: BlockingRecv error: recv got EOF from 127.0.0.1:60168 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:21:09.813378 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 23225
I20260926 02:21:09.816994 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:09.814374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.12.244.193:37985)
0926 02:21:09.814857 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:09.814864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:09.814891 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:21:09.814960 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:09.814966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:09.815287 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:09.815423 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:09.816261 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:09.816267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:09.816282 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:09.816303 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:09.816345 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:09.816359 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:09.816501 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:09.816570 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 02:21:09.816689 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260926 02:21:09.818289 23348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:09.899991 23353 heartbeater.cc:503] Master 127.12.244.254:44593 was elected leader, sending a full tablet report...
I20260926 02:21:09.901051 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 23225
W20260926 02:21:09.917955 23151 connection.cc:535] server connection from 127.12.244.193:35005 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 02:21:09.918217 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 23357
I20260926 02:21:09.921923 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:09.919191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.12.244.194:44991)
0926 02:21:09.919677 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:09.919685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:09.919714 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:21:09.919790 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:09.919798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:09.920025 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:09.920172 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:09.921106 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:09.921113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:09.921126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:09.921139 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:09.921181 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:09.921200 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:09.921332 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:09.921477 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:21:09.921600 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260926 02:21:09.923385 23480 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:10.004745 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 23357
I20260926 02:21:10.021046 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 23489
I20260926 02:21:10.025321 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:10.022108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.12.244.195:45401)
0926 02:21:10.022844 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:10.022851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:10.022893 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:21:10.022962 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:10.022969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:10.023265 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:10.023402 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:10.024496 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:10.024503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:10.024524 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:21:10.024542 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:10.024598 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:21:10.024613 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:10.024759 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:10.024815 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 02:21:10.024939 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":311,"threads_started":1}
I20260926 02:21:10.026595 23611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:10.108897 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 23489
I20260926 02:21:10.125289 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 23135
I20260926 02:21:10.126571 23196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:10.228178 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 23135
2026-09-26T02:21:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2507 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:21:10 dist-test-slave-qfmw krb5kdc[23635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:21:10 dist-test-slave-qfmw krb5kdc[23635](info): set up 2 sockets
Sep 26 02:21:10 dist-test-slave-qfmw krb5kdc[23635](info): commencing operation
krb5kdc: starting...
W20260926 02:21:11.342474 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:21:11 dist-test-slave-qfmw krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:21:11.589217 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45375
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:45375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:21:11.933667 23648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:11.933992 23648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:11.934199 23648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:11.941749 23648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:21:11.941874 23648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:11.941967 23648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:11.942070 23648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:21:11.942152 23648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:21:11.952513 23648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:45375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:11.953989 23648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:11.955485 23648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:11.967990 23653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:11.968076 23654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:11.968076 23656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:11.969280 23648 server_base.cc:1034] running on GCE node
I20260926 02:21:11.970314 23648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:11.972133 23648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:11.973383 23648 hybrid_clock.cc:648] HybridClock initialized: now 1790389271973337 us; error 52 us; skew 500 ppm
I20260926 02:21:11.973805 23648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:21:11 dist-test-slave-qfmw krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389271, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:21:11.981384 23648 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:21:11.983331 23648 webserver.cc:460] Webserver started at http://127.12.244.254:46001/ using document root <none> and password file <none>
I20260926 02:21:11.984093 23648 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:11.984251 23648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:11.984571 23648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:11.987800 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "b00618746e7044b6b63d928817767b02"
format_stamp: "Formatted at 2026-09-26 02:21:11 on dist-test-slave-qfmw"
server_key: "c2fbb0540abd8d5038cf213d58f2eec0"
server_key_iv: "282850a3aee5db3cdbd2c09d3178086f"
server_key_version: "encryptionkey@0"
I20260926 02:21:11.988601 23648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "b00618746e7044b6b63d928817767b02"
format_stamp: "Formatted at 2026-09-26 02:21:11 on dist-test-slave-qfmw"
server_key: "c2fbb0540abd8d5038cf213d58f2eec0"
server_key_iv: "282850a3aee5db3cdbd2c09d3178086f"
server_key_version: "encryptionkey@0"
I20260926 02:21:11.994235 23648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:21:11.998114 23663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:11.999794 23648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:21:12.000002 23648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "b00618746e7044b6b63d928817767b02"
format_stamp: "Formatted at 2026-09-26 02:21:11 on dist-test-slave-qfmw"
server_key: "c2fbb0540abd8d5038cf213d58f2eec0"
server_key_iv: "282850a3aee5db3cdbd2c09d3178086f"
server_key_version: "encryptionkey@0"
I20260926 02:21:12.000216 23648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:12.020463 23648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:12.024922 23648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:12.025239 23648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:12.044833 23648 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:45375
I20260926 02:21:12.044891 23714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:45375 every 8 connection(s)
I20260926 02:21:12.046320 23648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:21:12.050347 23715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:21:12.053465 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 23648
I20260926 02:21:12.053699 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:21:12.054510 13267 external_mini_cluster.cc:1428] Setting key e8d19a7e2097a77a12e50b1772d8c4ea
I20260926 02:21:12.063359 23715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02: Bootstrap starting.
I20260926 02:21:12.066588 23715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02: Neither blocks nor log segments found. Creating new log.
I20260926 02:21:12.067847 23715 log.cc:824] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:21:12 dist-test-slave-qfmw krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:21:12.071538 23715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02: No bootstrap required, opened a new log
I20260926 02:21:12.079208 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:12.056983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.12.244.254:45375)
0926 02:21:12.057850 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:12.057864 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:12.057909 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:21:12.059776 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:12.059812 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:12.060318 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:12.060773 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:12.062351 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:12.063863 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:12.065222 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:12.065944 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:12.066156 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:12.070896 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:12.070909 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:12.070932 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:12.070981 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:12.074534 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:12.075445 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:12.075452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:12.075459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:12.075640 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:12.076061 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:12.076066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:12.076071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:12.076701 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:12.077054 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:12.077529 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 02:21:12.078013 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":362,"threads_started":1}
I20260926 02:21:12.080003 23715 raft_consensus.cc:348] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00618746e7044b6b63d928817767b02" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45375 } }
I20260926 02:21:12.080428 23715 raft_consensus.cc:374] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:21:12.080546 23715 raft_consensus.cc:729] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00618746e7044b6b63d928817767b02, State: Initialized, Role: FOLLOWER
I20260926 02:21:12.081291 23715 consensus_queue.cc:260] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00618746e7044b6b63d928817767b02" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45375 } }
I20260926 02:21:12.081563 23715 raft_consensus.cc:388] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:21:12.081713 23715 raft_consensus.cc:482] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:21:12.081887 23715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:21:12.084815 23715 raft_consensus.cc:504] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00618746e7044b6b63d928817767b02" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45375 } }
I20260926 02:21:12.085395 23715 leader_election.cc:302] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b00618746e7044b6b63d928817767b02; no voters:
I20260926 02:21:12.085951 23715 leader_election.cc:288] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:21:12.086112 23720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:21:12.086850 23720 raft_consensus.cc:686] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [term 1 LEADER]: Becoming Leader. State: Replica: b00618746e7044b6b63d928817767b02, State: Running, Role: LEADER
I20260926 02:21:12.087425 23720 consensus_queue.cc:237] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00618746e7044b6b63d928817767b02" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45375 } }
I20260926 02:21:12.088377 23715 sys_catalog.cc:564] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:21:12.090708 23721 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b00618746e7044b6b63d928817767b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00618746e7044b6b63d928817767b02" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45375 } } }
I20260926 02:21:12.091151 23721 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [sys.catalog]: This master's current role is: LEADER
I20260926 02:21:12.090885 23722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b00618746e7044b6b63d928817767b02. Latest consensus state: current_term: 1 leader_uuid: "b00618746e7044b6b63d928817767b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00618746e7044b6b63d928817767b02" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45375 } } }
I20260926 02:21:12.091324 23722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02 [sys.catalog]: This master's current role is: LEADER
I20260926 02:21:12.093631 23725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:21:12.099215 23725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:21:12.108444 23725 catalog_manager.cc:1347] Generated new cluster ID: 37a34e7d94044e8e90cd0cbca35f5011
I20260926 02:21:12.108582 23725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:21:12.139338 23725 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:21:12.140748 23725 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:21:12.150629 23725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02: Generated new TSK 0
I20260926 02:21:12.151646 23725 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:21:12.215371 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45375
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:21:12.558112 23741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:12.558420 23741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:12.558516 23741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:12.566026 23741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:12.566157 23741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:12.566272 23741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:21:12.576469 23741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:12.577838 23741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:12.579597 23741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:12.594534 23749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:12.594617 23747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:12.594579 23746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:12.595625 23741 server_base.cc:1034] running on GCE node
I20260926 02:21:12.596259 23741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:12.597226 23741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:12.598441 23741 hybrid_clock.cc:648] HybridClock initialized: now 1790389272598368 us; error 64 us; skew 500 ppm
I20260926 02:21:12.598788 23741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:21:12 dist-test-slave-qfmw krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:21:12.602440 23741 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:21:12.604185 23741 webserver.cc:460] Webserver started at http://127.12.244.193:44597/ using document root <none> and password file <none>
I20260926 02:21:12.604943 23741 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:12.605050 23741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:12.605409 23741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:12.608620 23741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "b9f4eb8b99b448d79972da593581b196"
format_stamp: "Formatted at 2026-09-26 02:21:12 on dist-test-slave-qfmw"
server_key: "af15f5ad3e22d72f62d6aacc1aa005ea"
server_key_iv: "2125c822a4e715282e7166e2b7382c6c"
server_key_version: "encryptionkey@0"
I20260926 02:21:12.609468 23741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "b9f4eb8b99b448d79972da593581b196"
format_stamp: "Formatted at 2026-09-26 02:21:12 on dist-test-slave-qfmw"
server_key: "af15f5ad3e22d72f62d6aacc1aa005ea"
server_key_iv: "2125c822a4e715282e7166e2b7382c6c"
server_key_version: "encryptionkey@0"
I20260926 02:21:12.615211 23741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:21:12.618968 23756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:12.620580 23741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:21:12.620769 23741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "b9f4eb8b99b448d79972da593581b196"
format_stamp: "Formatted at 2026-09-26 02:21:12 on dist-test-slave-qfmw"
server_key: "af15f5ad3e22d72f62d6aacc1aa005ea"
server_key_iv: "2125c822a4e715282e7166e2b7382c6c"
server_key_version: "encryptionkey@0"
I20260926 02:21:12.620970 23741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:12.647043 23741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:12.651697 23741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:12.652014 23741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:12.653059 23741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:12.654448 23741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:12.654526 23741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:12.654693 23741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:12.654764 23741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:12.688592 23741 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:44681
I20260926 02:21:12.688663 23869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:44681 every 8 connection(s)
I20260926 02:21:12.690112 23741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:21:12.699450 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 23741
I20260926 02:21:12.699772 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:21:12.700644 13267 external_mini_cluster.cc:1428] Setting key 853fdf871408fd0548fc80e6308a2fc0
Sep 26 02:21:12 dist-test-slave-qfmw krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389272, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:21:12.717156 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:12.692897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:41393 (local address 127.12.244.254:45375)
0926 02:21:12.693361 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:12.693366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:12.694816 (+ 1450us) server_negotiation.cc:407] Connection header received
0926 02:21:12.698163 (+ 3347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:12.698185 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:12.698301 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:12.698414 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:12.700986 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:12.702157 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:12.705026 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:12.705479 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:12.705595 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:12.710400 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:12.710405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:12.710409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:12.710449 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:12.713351 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:12.714367 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:12.714373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:12.714375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:12.714519 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:12.715048 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:12.715053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:12.715055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:12.715448 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:12.715639 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:12.716485 (+ 846us) server_negotiation.cc:299] Negotiation successful
0926 02:21:12.716746 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":214,"threads_started":1}
I20260926 02:21:12.718298 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:12.693554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45375 (local address 127.12.244.193:41393)
0926 02:21:12.694625 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:12.694690 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:12.697730 (+ 3040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:12.698742 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:12.698762 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:12.699761 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:12.700737 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:12.700767 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:12.702376 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:12.702382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:12.704747 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:12.704770 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:12.705751 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:12.705762 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:12.706157 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:12.707026 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:12.707077 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:12.710092 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:12.713573 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:21:12.713584 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:21:12.713676 (+ 92us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:21:12.714191 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:21:12.714657 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:21:12.714663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:21:12.714667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:21:12.714897 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:21:12.715616 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:12.715628 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:12.716047 (+ 419us) client_negotiation.cc:769] Sending connection context
0926 02:21:12.716356 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 02:21:12.717007 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":400,"threads_started":1}
I20260926 02:21:12.720304 23870 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45375
I20260926 02:21:12.720665 23870 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:12.721643 23870 heartbeater.cc:511] Master 127.12.244.254:45375 requested a full tablet report, sending...
I20260926 02:21:12.724686 23680 ts_manager.cc:194] Registered new tserver with Master: b9f4eb8b99b448d79972da593581b196 (127.12.244.193:44681)
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:21:12.726902 23680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:41393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:21:12.758850 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45375
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:21:13.149041 23876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:13.149341 23876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:13.149499 23876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:13.157131 23876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:13.157269 23876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:13.157450 23876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:21:13.167519 23876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:13.168912 23876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:13.170272 23876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:13.184423 23882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:13.184494 23881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:13.184746 23884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:13.185369 23876 server_base.cc:1034] running on GCE node
I20260926 02:21:13.186018 23876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:13.187150 23876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:13.188391 23876 hybrid_clock.cc:648] HybridClock initialized: now 1790389273188341 us; error 48 us; skew 500 ppm
I20260926 02:21:13.188794 23876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:21:13 dist-test-slave-qfmw krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389273, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:21:13.192368 23876 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:21:13.194154 23876 webserver.cc:460] Webserver started at http://127.12.244.194:33751/ using document root <none> and password file <none>
I20260926 02:21:13.194890 23876 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:13.195034 23876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:13.195425 23876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:13.198396 23876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "17294a9dcf7e44a4904495b14f8588bc"
format_stamp: "Formatted at 2026-09-26 02:21:13 on dist-test-slave-qfmw"
server_key: "2afd614f2734b347a33d308d5c52cd58"
server_key_iv: "683bf5a0558f06e19da42fe7c2f2224c"
server_key_version: "encryptionkey@0"
I20260926 02:21:13.199215 23876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "17294a9dcf7e44a4904495b14f8588bc"
format_stamp: "Formatted at 2026-09-26 02:21:13 on dist-test-slave-qfmw"
server_key: "2afd614f2734b347a33d308d5c52cd58"
server_key_iv: "683bf5a0558f06e19da42fe7c2f2224c"
server_key_version: "encryptionkey@0"
I20260926 02:21:13.204622 23876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:21:13.208334 23891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:13.209851 23876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 02:21:13.209987 23876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "17294a9dcf7e44a4904495b14f8588bc"
format_stamp: "Formatted at 2026-09-26 02:21:13 on dist-test-slave-qfmw"
server_key: "2afd614f2734b347a33d308d5c52cd58"
server_key_iv: "683bf5a0558f06e19da42fe7c2f2224c"
server_key_version: "encryptionkey@0"
I20260926 02:21:13.210167 23876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:13.231940 23876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:13.236258 23876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:13.236542 23876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:13.237476 23876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:13.238798 23876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:13.238869 23876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:13.238962 23876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:13.239006 23876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:13.273859 23876 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:37319
I20260926 02:21:13.273993 24004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:37319 every 8 connection(s)
I20260926 02:21:13.275251 23876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:21:13.284582 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 23876
I20260926 02:21:13.284781 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:21:13.285198 13267 external_mini_cluster.cc:1428] Setting key 00d74b650d1e996d89171aa77678e772
Sep 26 02:21:13 dist-test-slave-qfmw krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389273, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:21:13.298066 24007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.277932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:53653 (local address 127.12.244.254:45375)
0926 02:21:13.278379 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:13.278383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:13.279417 (+ 1034us) server_negotiation.cc:407] Connection header received
0926 02:21:13.281639 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:13.281644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:13.281747 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:13.281868 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:13.283580 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.284337 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.286478 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.287032 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.287236 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.291709 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:13.291715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:13.291719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:13.291756 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:13.294639 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:13.295625 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:13.295629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:13.295631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:13.295772 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:13.296230 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:13.296234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:13.296236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:13.296527 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:13.296683 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:13.297434 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 02:21:13.297665 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20260926 02:21:13.299170 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.278447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45375 (local address 127.12.244.194:53653)
0926 02:21:13.279254 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:13.279300 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:13.281327 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:13.282131 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:13.282140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:13.282683 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:13.283396 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.283418 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.284504 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.284509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.286299 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.286314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.287339 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.287349 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.287656 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.288478 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:13.288517 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:13.291431 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:13.294809 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:21:13.294820 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:21:13.294842 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:21:13.295494 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:21:13.295892 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:21:13.295897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:21:13.295901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:21:13.296102 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:21:13.296664 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:13.296672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:13.297054 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 02:21:13.297335 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 02:21:13.297925 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":304,"threads_started":1}
I20260926 02:21:13.300688 24005 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45375
I20260926 02:21:13.301048 24005 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:13.301807 24005 heartbeater.cc:511] Master 127.12.244.254:45375 requested a full tablet report, sending...
I20260926 02:21:13.303457 23680 ts_manager.cc:194] Registered new tserver with Master: 17294a9dcf7e44a4904495b14f8588bc (127.12.244.194:37319)
I20260926 02:21:13.304801 23680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:53653
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:21:13.339273 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45375
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:21:13.679970 24011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:21:13.680272 24011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:21:13.680441 24011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:21:13.687837 24011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:21:13.688019 24011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:21:13.688201 24011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:21:13.698758 24011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:21:13.700131 24011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:21:13.701551 24011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:21:13.715590 24016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:13.715762 24019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:21:13.715794 24017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:21:13.716507 24011 server_base.cc:1034] running on GCE node
I20260926 02:21:13.717219 24011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:21:13.718346 24011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:21:13.719594 24011 hybrid_clock.cc:648] HybridClock initialized: now 1790389273719534 us; error 49 us; skew 500 ppm
I20260926 02:21:13.719951 24011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:21:13 dist-test-slave-qfmw krb5kdc[23635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389273, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:21:13.723523 24011 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:21:13.725131 24011 webserver.cc:460] Webserver started at http://127.12.244.195:34101/ using document root <none> and password file <none>
I20260926 02:21:13.725903 24011 fs_manager.cc:359] Metadata directory not provided
I20260926 02:21:13.726012 24011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:21:13.726287 24011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:21:13.729440 24011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "3885cf1f41dc481f9d653e9f448716af"
format_stamp: "Formatted at 2026-09-26 02:21:13 on dist-test-slave-qfmw"
server_key: "a8e0f8ae2f9d15b5448c013489176975"
server_key_iv: "782b33d22a8a578fe24fa2d91f5cbadb"
server_key_version: "encryptionkey@0"
I20260926 02:21:13.730290 24011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "3885cf1f41dc481f9d653e9f448716af"
format_stamp: "Formatted at 2026-09-26 02:21:13 on dist-test-slave-qfmw"
server_key: "a8e0f8ae2f9d15b5448c013489176975"
server_key_iv: "782b33d22a8a578fe24fa2d91f5cbadb"
server_key_version: "encryptionkey@0"
I20260926 02:21:13.730521 23870 heartbeater.cc:503] Master 127.12.244.254:45375 was elected leader, sending a full tablet report...
I20260926 02:21:13.736421 24011 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:21:13.740372 24026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:13.742072 24011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:21:13.742275 24011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "3885cf1f41dc481f9d653e9f448716af"
format_stamp: "Formatted at 2026-09-26 02:21:13 on dist-test-slave-qfmw"
server_key: "a8e0f8ae2f9d15b5448c013489176975"
server_key_iv: "782b33d22a8a578fe24fa2d91f5cbadb"
server_key_version: "encryptionkey@0"
I20260926 02:21:13.742466 24011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:21:13.758651 24011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:21:13.763520 24011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:21:13.763850 24011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:21:13.764930 24011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:21:13.766376 24011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:21:13.766448 24011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:13.766601 24011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:21:13.766656 24011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:21:13.801724 24011 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:37541
I20260926 02:21:13.801819 24139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:37541 every 8 connection(s)
I20260926 02:21:13.803431 24011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:21:13.813450 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 24011
I20260926 02:21:13.813669 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:21:13.814133 13267 external_mini_cluster.cc:1428] Setting key 82cad28405b73f9f6ea62b1ea33d435f
Sep 26 02:21:13 dist-test-slave-qfmw krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389273, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:21:13.825603 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.806969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:57321 (local address 127.12.244.254:45375)
0926 02:21:13.807479 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:13.807483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:13.808497 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 02:21:13.810403 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:13.810409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:13.810514 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:13.810652 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:13.812370 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.813159 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.815318 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.815898 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.816040 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.819770 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:13.819781 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:13.819784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:13.819821 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:13.822199 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:13.823022 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:13.823027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:13.823031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:13.823231 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:13.823690 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:13.823694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:13.823698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:13.823976 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:13.824145 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:13.824959 (+ 814us) server_negotiation.cc:299] Negotiation successful
0926 02:21:13.825200 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":299,"threads_started":1}
I20260926 02:21:13.826669 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.807579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45375 (local address 127.12.244.195:57321)
0926 02:21:13.808330 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:13.808378 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:13.810110 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:13.810918 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:13.810927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:13.811551 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:21:13.812172 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.812190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.813337 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.813341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.815158 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.815175 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.816140 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.816147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.816417 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.817177 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:21:13.817202 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:21:13.819521 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:21:13.822387 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:21:13.822396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:21:13.822418 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:21:13.822884 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:21:13.823345 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:21:13.823350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:21:13.823354 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:21:13.823559 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:21:13.824129 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:21:13.824137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:21:13.824566 (+ 429us) client_negotiation.cc:769] Sending connection context
0926 02:21:13.824852 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 02:21:13.825437 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":293,"threads_started":1}
I20260926 02:21:13.828233 24140 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45375
I20260926 02:21:13.828593 24140 heartbeater.cc:464] Registering TS with master...
I20260926 02:21:13.829516 24140 heartbeater.cc:511] Master 127.12.244.254:45375 requested a full tablet report, sending...
I20260926 02:21:13.830952 23680 ts_manager.cc:194] Registered new tserver with Master: 3885cf1f41dc481f9d653e9f448716af (127.12.244.195:37541)
I20260926 02:21:13.831867 23680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:57321
I20260926 02:21:13.842505 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:21:13.859494 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.847312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.12.244.254:45375)
0926 02:21:13.847555 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:13.847562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:13.847904 (+ 342us) server_negotiation.cc:407] Connection header received
0926 02:21:13.848048 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:13.848052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:13.848134 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:13.848218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:13.849678 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.850413 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.851744 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.852155 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.852264 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.854473 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:13.854478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:13.854482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:13.854518 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:13.856991 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:13.857624 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:13.857628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:13.857630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:13.857781 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:13.858085 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:13.858087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:13.858089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:13.858405 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:13.858579 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:13.858911 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 02:21:13.859075 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:21:13.863626 23680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:21:13.870491 23680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:21:13.894200 24153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.883550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:37319 (local address 127.0.0.1:40926)
0926 02:21:13.884902 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:13.884944 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:13.885115 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:13.886182 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:13.886189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:13.886249 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:21:13.886661 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.886675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.888560 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.888566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.891303 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.891318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.893065 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.893074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.893297 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.893360 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 02:21:13.893487 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 02:21:13.893681 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":20,"spinlock_wait_cycles":18688,"thread_start_us":939,"threads_started":1}
I20260926 02:21:13.894984 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.883874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:37541 (local address 127.0.0.1:58744)
0926 02:21:13.886099 (+ 2225us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:13.886118 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:13.886243 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:13.887330 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:13.887338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:13.887373 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:21:13.887815 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.887826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.889877 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.889889 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.892120 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.892132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.894036 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.894042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.894226 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.894259 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:21:13.894342 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 02:21:13.894480 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2090,"mutex_wait_us":23,"spinlock_wait_cycles":1792,"thread_start_us":1958,"threads_started":1}
I20260926 02:21:13.895924 24157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.886205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44681 (local address 127.0.0.1:41894)
0926 02:21:13.888550 (+ 2345us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:13.888567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:13.888714 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:13.889267 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:13.889272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:13.889294 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:21:13.889762 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.889775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.891548 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.891554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.893166 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:13.893177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.895138 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.895146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:13.895334 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.895369 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:21:13.895473 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:21:13.895596 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2213,"thread_start_us":2023,"threads_started":1}
I20260926 02:21:13.896148 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.886755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.12.244.193:44681)
0926 02:21:13.888089 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:13.888098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:13.888670 (+ 572us) server_negotiation.cc:407] Connection header received
0926 02:21:13.888885 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:13.888891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:13.889075 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:13.889256 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:21:13.889936 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.891411 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.893330 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.894606 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.894791 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.894972 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:13.895519 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 02:21:13.895678 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1178,"thread_start_us":987,"threads_started":1}
I20260926 02:21:13.896435 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.884844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.12.244.195:37541)
0926 02:21:13.885716 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:13.885723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:13.886395 (+ 672us) server_negotiation.cc:407] Connection header received
0926 02:21:13.886504 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:13.886512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:13.886779 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:13.886979 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:21:13.888011 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.889675 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.892308 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.893829 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.894139 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.894566 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:13.894644 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 02:21:13.894877 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":649,"threads_started":1}
I20260926 02:21:13.897101 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:13.884191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.12.244.194:37319)
0926 02:21:13.885542 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:13.885553 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:13.885581 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:21:13.885690 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:13.885697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:13.885964 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:13.886181 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:21:13.886875 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.888410 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.891478 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:13.892814 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:13.896406 (+ 3592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:13.896589 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:13.896665 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 02:21:13.896825 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":1064,"threads_started":1}
I20260926 02:21:13.899765 23805 tablet_service.cc:1467] Processing CreateTablet for tablet 640c4b408d1448ac8b2d88370b05f605 (DEFAULT_TABLE table=test-table [id=a08d836283134a7bac09f60472a842de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:21:13.900539 24074 tablet_service.cc:1467] Processing CreateTablet for tablet 640c4b408d1448ac8b2d88370b05f605 (DEFAULT_TABLE table=test-table [id=a08d836283134a7bac09f60472a842de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:21:13.900539 23940 tablet_service.cc:1467] Processing CreateTablet for tablet 640c4b408d1448ac8b2d88370b05f605 (DEFAULT_TABLE table=test-table [id=a08d836283134a7bac09f60472a842de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:21:13.901733 23805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640c4b408d1448ac8b2d88370b05f605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:21:13.901876 23940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640c4b408d1448ac8b2d88370b05f605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:21:13.901875 24074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640c4b408d1448ac8b2d88370b05f605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:21:13.916498 24159 tablet_bootstrap.cc:492] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Bootstrap starting.
I20260926 02:21:13.917573 24160 tablet_bootstrap.cc:492] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Bootstrap starting.
I20260926 02:21:13.920611 24159 tablet_bootstrap.cc:654] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Neither blocks nor log segments found. Creating new log.
I20260926 02:21:13.921352 24160 tablet_bootstrap.cc:654] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Neither blocks nor log segments found. Creating new log.
I20260926 02:21:13.922049 24159 log.cc:824] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Log is configured to *not* fsync() on all Append() calls
I20260926 02:21:13.922639 24160 log.cc:824] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Log is configured to *not* fsync() on all Append() calls
I20260926 02:21:13.925340 24161 tablet_bootstrap.cc:492] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Bootstrap starting.
I20260926 02:21:13.926137 24159 tablet_bootstrap.cc:492] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: No bootstrap required, opened a new log
I20260926 02:21:13.926429 24159 ts_tablet_manager.cc:1397] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:21:13.926859 24160 tablet_bootstrap.cc:492] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: No bootstrap required, opened a new log
I20260926 02:21:13.927193 24160 ts_tablet_manager.cc:1397] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260926 02:21:13.929239 24161 tablet_bootstrap.cc:654] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Neither blocks nor log segments found. Creating new log.
I20260926 02:21:13.930649 24161 log.cc:824] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Log is configured to *not* fsync() on all Append() calls
I20260926 02:21:13.935153 24161 tablet_bootstrap.cc:492] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: No bootstrap required, opened a new log
I20260926 02:21:13.935518 24161 ts_tablet_manager.cc:1397] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.007s
I20260926 02:21:13.935256 24160 raft_consensus.cc:348] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:13.935793 24160 raft_consensus.cc:374] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:21:13.935894 24160 raft_consensus.cc:729] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17294a9dcf7e44a4904495b14f8588bc, State: Initialized, Role: FOLLOWER
I20260926 02:21:13.936554 24160 consensus_queue.cc:260] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:13.937269 24159 raft_consensus.cc:348] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:13.937718 24159 raft_consensus.cc:374] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:21:13.937834 24159 raft_consensus.cc:729] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f4eb8b99b448d79972da593581b196, State: Initialized, Role: FOLLOWER
I20260926 02:21:13.938540 24005 heartbeater.cc:503] Master 127.12.244.254:45375 was elected leader, sending a full tablet report...
I20260926 02:21:13.938549 24159 consensus_queue.cc:260] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:13.939342 24160 ts_tablet_manager.cc:1428] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260926 02:21:13.941414 24159 ts_tablet_manager.cc:1428] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
W20260926 02:21:13.942776 23871 tablet.cc:2369] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:21:13.947626 24161 raft_consensus.cc:348] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:13.948120 24161 raft_consensus.cc:374] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:21:13.948230 24161 raft_consensus.cc:729] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3885cf1f41dc481f9d653e9f448716af, State: Initialized, Role: FOLLOWER
I20260926 02:21:13.948952 24161 consensus_queue.cc:260] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:13.950079 24140 heartbeater.cc:503] Master 127.12.244.254:45375 was elected leader, sending a full tablet report...
I20260926 02:21:13.950862 24161 ts_tablet_manager.cc:1428] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260926 02:21:14.027521 24006 tablet.cc:2369] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:21:14.055570 24141 tablet.cc:2369] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:21:14.258293 24165 raft_consensus.cc:482] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:21:14.258661 24165 raft_consensus.cc:504] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:14.260519 24165 leader_election.cc:288] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3885cf1f41dc481f9d653e9f448716af (127.12.244.195:37541), b9f4eb8b99b448d79972da593581b196 (127.12.244.193:44681)
I20260926 02:21:14.268750 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.261672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:52299 (local address 127.12.244.193:44681)
0926 02:21:14.261889 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:14.261895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:14.262281 (+ 386us) server_negotiation.cc:407] Connection header received
0926 02:21:14.262405 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:14.262408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:14.262465 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:14.262580 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:21:14.263127 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.264033 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.266399 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.267634 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.267774 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.267844 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:14.268346 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 02:21:14.268482 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:21:14.268800 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.260930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:38587 (local address 127.12.244.195:37541)
0926 02:21:14.261182 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:14.261189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:14.261500 (+ 311us) server_negotiation.cc:407] Connection header received
0926 02:21:14.261728 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:14.261738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:14.261887 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:14.262004 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:21:14.262598 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.263619 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.266227 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.267565 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.267767 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.267844 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:14.268346 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 02:21:14.268529 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 02:21:14.269054 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.260788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:37541 (local address 127.12.244.194:38587)
0926 02:21:14.261349 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:14.261365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:14.261552 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:14.262038 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:14.262043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:14.262073 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:21:14.262451 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:14.262460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.263756 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.263760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:14.266014 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:14.266030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.267860 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.267866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:14.268185 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.268210 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:21:14.268290 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:21:14.268486 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":116,"thread_start_us":297,"threads_started":1}
I20260926 02:21:14.269093 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.261513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:44681 (local address 127.12.244.194:52299)
0926 02:21:14.262136 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 02:21:14.262155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:21:14.262287 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:21:14.262615 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:21:14.262618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:21:14.262642 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:21:14.262932 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:14.262940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.264160 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.264164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:14.266249 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:21:14.266258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.267869 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.267873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:21:14.268147 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.268189 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:21:14.268290 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:21:14.268486 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":71,"thread_start_us":293,"threads_started":1}
I20260926 02:21:14.270210 24094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c4b408d1448ac8b2d88370b05f605" candidate_uuid: "17294a9dcf7e44a4904495b14f8588bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3885cf1f41dc481f9d653e9f448716af" is_pre_election: true
I20260926 02:21:14.270210 23825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c4b408d1448ac8b2d88370b05f605" candidate_uuid: "17294a9dcf7e44a4904495b14f8588bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f4eb8b99b448d79972da593581b196" is_pre_election: true
I20260926 02:21:14.270740 24094 raft_consensus.cc:2454] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17294a9dcf7e44a4904495b14f8588bc in term 0.
I20260926 02:21:14.270761 23825 raft_consensus.cc:2454] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17294a9dcf7e44a4904495b14f8588bc in term 0.
I20260926 02:21:14.271462 23895 leader_election.cc:302] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17294a9dcf7e44a4904495b14f8588bc, 3885cf1f41dc481f9d653e9f448716af; no voters:
I20260926 02:21:14.271911 24165 raft_consensus.cc:2781] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:21:14.272083 24165 raft_consensus.cc:482] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:21:14.272197 24165 raft_consensus.cc:3037] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:21:14.274644 24165 raft_consensus.cc:504] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:14.275558 24165 leader_election.cc:288] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [CANDIDATE]: Term 1 election: Requested vote from peers 3885cf1f41dc481f9d653e9f448716af (127.12.244.195:37541), b9f4eb8b99b448d79972da593581b196 (127.12.244.193:44681)
I20260926 02:21:14.276022 24094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c4b408d1448ac8b2d88370b05f605" candidate_uuid: "17294a9dcf7e44a4904495b14f8588bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3885cf1f41dc481f9d653e9f448716af"
I20260926 02:21:14.276278 24094 raft_consensus.cc:3037] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:21:14.276268 23825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640c4b408d1448ac8b2d88370b05f605" candidate_uuid: "17294a9dcf7e44a4904495b14f8588bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f4eb8b99b448d79972da593581b196"
I20260926 02:21:14.276474 23825 raft_consensus.cc:3037] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:21:14.279781 24094 raft_consensus.cc:2454] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17294a9dcf7e44a4904495b14f8588bc in term 1.
I20260926 02:21:14.279781 23825 raft_consensus.cc:2454] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17294a9dcf7e44a4904495b14f8588bc in term 1.
I20260926 02:21:14.280268 23895 leader_election.cc:302] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17294a9dcf7e44a4904495b14f8588bc, b9f4eb8b99b448d79972da593581b196; no voters:
I20260926 02:21:14.280603 24165 raft_consensus.cc:2781] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:21:14.281078 24165 raft_consensus.cc:686] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 1 LEADER]: Becoming Leader. State: Replica: 17294a9dcf7e44a4904495b14f8588bc, State: Running, Role: LEADER
I20260926 02:21:14.281611 24165 consensus_queue.cc:237] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } }
I20260926 02:21:14.286965 23680 catalog_manager.cc:5495] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 17294a9dcf7e44a4904495b14f8588bc (127.12.244.194). New cstate: current_term: 1 leader_uuid: "17294a9dcf7e44a4904495b14f8588bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17294a9dcf7e44a4904495b14f8588bc" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 } health_report { overall_health: UNKNOWN } } }
I20260926 02:21:14.352844 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.347714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.12.244.194:37319)
0926 02:21:14.347889 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:14.347894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:14.348041 (+ 147us) server_negotiation.cc:407] Connection header received
0926 02:21:14.348201 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:14.348205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:14.348316 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:14.348444 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:21:14.348756 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.349444 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.350713 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.351204 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.351303 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.351722 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:21:14.352277 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:21:14.352407 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:14.352556 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 02:21:14.352625 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:21:14.361502 23825 raft_consensus.cc:1261] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 1 FOLLOWER]: Refusing update from remote peer 17294a9dcf7e44a4904495b14f8588bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:21:14.361506 24094 raft_consensus.cc:1261] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 1 FOLLOWER]: Refusing update from remote peer 17294a9dcf7e44a4904495b14f8588bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:21:14.362552 24170 consensus_queue.cc:1035] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3885cf1f41dc481f9d653e9f448716af" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 37541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:21:14.363021 24165 consensus_queue.cc:1035] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f4eb8b99b448d79972da593581b196" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 44681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:21:14.395375 23680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:21:14.395715 23680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:21:14.401084 23680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b00618746e7044b6b63d928817767b02: Sending DeleteTablet for 3 replicas of tablet 640c4b408d1448ac8b2d88370b05f605
I20260926 02:21:14.402355 23940 tablet_service.cc:1514] Processing DeleteTablet for tablet 640c4b408d1448ac8b2d88370b05f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:21:14 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:40926
I20260926 02:21:14.402690 24074 tablet_service.cc:1514] Processing DeleteTablet for tablet 640c4b408d1448ac8b2d88370b05f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:21:14 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:58744
I20260926 02:21:14.403445 23805 tablet_service.cc:1514] Processing DeleteTablet for tablet 640c4b408d1448ac8b2d88370b05f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:21:14 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41894
I20260926 02:21:14.404174 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 23741
I20260926 02:21:14.404608 24073 tablet_service.cc:1514] Processing DeleteTablet for tablet 640c4b408d1448ac8b2d88370b05f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:21:14 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:58744
I20260926 02:21:14.404845 23804 tablet_service.cc:1514] Processing DeleteTablet for tablet 640c4b408d1448ac8b2d88370b05f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:21:14 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41894
I20260926 02:21:14.407644 24185 tablet_replica.cc:330] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: stopping tablet replica
I20260926 02:21:14.408138 24185 raft_consensus.cc:2229] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:21:14.408737 24185 raft_consensus.cc:2258] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:21:14.411367 24185 ts_tablet_manager.cc:1905] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:21:14.415004 24183 tablet_replica.cc:330] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: stopping tablet replica
I20260926 02:21:14.415524 24183 raft_consensus.cc:2229] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:21:14.416508 24183 raft_consensus.cc:2258] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:21:14.419648 24185 ts_tablet_manager.cc:1918] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:21:14.419893 24185 log.cc:1196] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/640c4b408d1448ac8b2d88370b05f605
I20260926 02:21:14.420522 24185 ts_tablet_manager.cc:1939] T 640c4b408d1448ac8b2d88370b05f605 P b9f4eb8b99b448d79972da593581b196: Deleting consensus metadata
I20260926 02:21:14.422632 23667 catalog_manager.cc:4841] TS b9f4eb8b99b448d79972da593581b196 (127.12.244.193:44681): tablet 640c4b408d1448ac8b2d88370b05f605 (table test-table [id=a08d836283134a7bac09f60472a842de]) successfully deleted
I20260926 02:21:14.423168 24184 tablet_replica.cc:330] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: stopping tablet replica
W20260926 02:21:14.423353 23667 catalog_manager.cc:4808] TS b9f4eb8b99b448d79972da593581b196 (127.12.244.193:44681): delete failed for tablet 640c4b408d1448ac8b2d88370b05f605 because the tablet was not found. No further retry: Not found: Tablet not found: 640c4b408d1448ac8b2d88370b05f605
I20260926 02:21:14.423657 24184 raft_consensus.cc:2229] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:21:14.424175 24184 raft_consensus.cc:2258] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:21:14.427163 24183 ts_tablet_manager.cc:1905] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:21:14.428023 24184 ts_tablet_manager.cc:1905] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:21:14.435209 24184 ts_tablet_manager.cc:1918] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:21:14.435212 24183 ts_tablet_manager.cc:1918] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:21:14.435402 24184 log.cc:1196] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/640c4b408d1448ac8b2d88370b05f605
I20260926 02:21:14.435403 24183 log.cc:1196] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/640c4b408d1448ac8b2d88370b05f605
Sep 26 02:21:14 dist-test-slave-qfmw krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:21:14.435981 24184 ts_tablet_manager.cc:1939] T 640c4b408d1448ac8b2d88370b05f605 P 3885cf1f41dc481f9d653e9f448716af: Deleting consensus metadata
I20260926 02:21:14.435987 24183 ts_tablet_manager.cc:1939] T 640c4b408d1448ac8b2d88370b05f605 P 17294a9dcf7e44a4904495b14f8588bc: Deleting consensus metadata
I20260926 02:21:14.437876 23667 catalog_manager.cc:4841] TS 3885cf1f41dc481f9d653e9f448716af (127.12.244.195:37541): tablet 640c4b408d1448ac8b2d88370b05f605 (table test-table [id=a08d836283134a7bac09f60472a842de]) successfully deleted
I20260926 02:21:14.438012 23664 catalog_manager.cc:4841] TS 17294a9dcf7e44a4904495b14f8588bc (127.12.244.194:37319): tablet 640c4b408d1448ac8b2d88370b05f605 (table test-table [id=a08d836283134a7bac09f60472a842de]) successfully deleted
W20260926 02:21:14.438782 23667 catalog_manager.cc:4808] TS 3885cf1f41dc481f9d653e9f448716af (127.12.244.195:37541): delete failed for tablet 640c4b408d1448ac8b2d88370b05f605 because the tablet was not found. No further retry: Not found: Tablet not found: 640c4b408d1448ac8b2d88370b05f605
I20260926 02:21:14.445170 24158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.405560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.12.244.193:44681)
0926 02:21:14.405744 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:14.405750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:14.419483 (+ 13733us) server_negotiation.cc:407] Connection header received
0926 02:21:14.420219 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:14.420229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:14.420337 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:14.420442 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:14.423293 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.424416 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.426684 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.427385 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.427529 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.437116 (+ 9587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:14.437137 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:14.437158 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:14.437217 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:14.441302 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:14.442388 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:14.442401 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:14.442407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:14.442599 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:14.443074 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:14.443079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:14.443083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:14.443599 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:14.443953 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:14.444510 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 02:21:14.444764 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:21:14.445952 23865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:14.540196 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 23741
I20260926 02:21:14.560577 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 23876
Sep 26 02:21:14 dist-test-slave-qfmw krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:21:14.579532 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.561434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.12.244.194:37319)
0926 02:21:14.561641 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:14.561646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:14.561795 (+ 149us) server_negotiation.cc:407] Connection header received
0926 02:21:14.561920 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:14.561925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:14.562016 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:14.562115 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:14.563681 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.564604 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.566509 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.567066 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.567293 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.572152 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:14.572171 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:14.572189 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:14.572247 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:14.576169 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:14.577098 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:14.577107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:14.577110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:14.577295 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:14.577735 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:14.577744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:14.577749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:14.578147 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:14.578531 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:14.578838 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 02:21:14.579085 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:21:14.580371 24000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:14.700134 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 23876
I20260926 02:21:14.722463 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 24011
Sep 26 02:21:14 dist-test-slave-qfmw krb5kdc[23635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:21:14.738441 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:21:14.723299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.12.244.195:37541)
0926 02:21:14.723474 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:21:14.723479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:21:14.723497 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:21:14.723652 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:21:14.723656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:21:14.723718 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:21:14.723847 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:21:14.725061 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.725755 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.726985 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:21:14.727601 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:21:14.727704 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:21:14.732235 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:21:14.732253 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:21:14.732271 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:21:14.732320 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:21:14.735420 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:14.736221 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:14.736230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:14.736232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:14.736389 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:21:14.736839 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:21:14.736843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:21:14.736845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:21:14.737232 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:21:14.737447 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:21:14.737827 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 02:21:14.738012 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:21:14.739382 24135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:14.830152 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 24011
I20260926 02:21:14.847043 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 23648
I20260926 02:21:14.848268 23710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:21:14.957845 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 23648
2026-09-26T02:21:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4731 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:21:15 dist-test-slave-qfmw krb5kdc[24193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:21:15 dist-test-slave-qfmw krb5kdc[24193](info): set up 2 sockets
Sep 26 02:21:15 dist-test-slave-qfmw krb5kdc[24193](info): commencing operation
krb5kdc: starting...
W20260926 02:21:16.074608 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:21:16 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:21:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/rangeradmin_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/rangeradmin_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/rangerlookup_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/rangerlookup_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
I20260926 02:21:16.430675 13267 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3VVuru/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/postgres -l logfile start
2026-09-26 02:21:19.416 UTC [24224] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:21:19.416 UTC [24224] LOG: listening on IPv4 address "127.12.244.212", port 53679
2026-09-26 02:21:19.422 UTC [24224] LOG: listening on Unix socket "/tmp/.s.PGSQL.53679"
2026-09-26 02:21:19.447 UTC [24227] LOG: database system was shut down at 2026-09-26 02:21:17 UTC
2026-09-26 02:21:19.453 UTC [24224] LOG: database system is ready to accept connections
I20260926 02:21:20.450124 13267 mini_postgres.cc:96] Postgres bound to 53679
2026-09-26 02:21:20.468 UTC [24235] FATAL: database "slave" does not exist
127.12.244.212:53679 - accepting connections
I20260926 02:21:20.469094 13267 mini_ranger.cc:161] Starting Ranger...
I20260926 02:21:20.496207 13267 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:21:20.998246 13267 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:21:20.998390 13267 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-admin
2026-09-26 02:21:21,299 [I] DB FLAVOR :POSTGRES
2026-09-26 02:21:21,299 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:21:21,299 [I] Checking connection..
2026-09-26 02:21:21,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:21:21,551 [I] Checking connection passed.
2026-09-26 02:21:21,551 [I] --------- Verifying version history table ---------
2026-09-26 02:21:21,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:21:21,792 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:21:21,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:21:22,047 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:21:22,047 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:21:22,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:21:22.283 UTC [24299] WARNING: there is no transaction in progress
2026-09-26 02:21:22,290 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:21:22,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:21:22,564 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:21:22,565 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:21:22,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:21:22,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:21:23,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qfmw.c.gcp-upstream.internal','N') ;"
2026-09-26 02:21:23,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:21:23,555 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:21:23,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:21:23,822 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:21:23,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:21:24,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:21:24,331 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:21:24,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:21:25.521 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.537 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.554 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.570 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.585 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.604 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.666 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.674 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.689 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.700 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.712 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.727 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.738 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.744 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.751 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:25.758 UTC [24409] WARNING: there is no transaction in progress
2026-09-26 02:21:26,516 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:21:26,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:21:26,773 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:21:26,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:21:27,045 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:21:27,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:21:27,294 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:21:27,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:21:27,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:21:27,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qfmw.c.gcp-upstream.internal';"
2026-09-26 02:21:28,081 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:21:28,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:21:28,322 [I] DB_PATCHES have already been applied
I20260926 02:21:28.329079 13267 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-admin:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3VVuru/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:21:28.329185 13267 mini_ranger.cc:193] Using host: 127.12.244.212
I20260926 02:21:28.345039 13267 mini_ranger.cc:227] Ranger admin URL: http://127.12.244.212:40857
Sep 26, 2026 2:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:21:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:21:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:21:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:21:29.557552 13267 mini_ranger.cc:160] Time spent starting Ranger: real 9.088s user 0.002s sys 0.042s
Sep 26, 2026 2:21:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:21:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:21:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:21:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:21:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:21:52.897110 13267 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/rangerkms_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/rangerkms_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.244.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.244.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:21:53 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389313, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:21:53.114269 13267 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:21:53.140501 13267 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:21:54.199321 13267 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:21:54.199506 13267 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms
I20260926 02:21:54.199642 13267 mini_ranger_kms.cc:227] Using postgres at 127.12.244.212:53679
2026-09-26 02:21:54,647 [I] DB FLAVOR :POSTGRES
2026-09-26 02:21:54,648 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:21:54,648 [I] Checking connection
2026-09-26 02:21:54,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:21:54,928 [I] connection success
2026-09-26 02:21:54,928 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:21:54,928 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:21:54,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:21:55,207 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:21:55,208 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:21:55,208 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:21:55,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:53679/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:21:55,496 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:21:56.045744 13267 mini_ranger_kms.cc:310] Created kms service
I20260926 02:21:56.265869 13267 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:21:56.313933 13267 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:21:56 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389313, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:21:56.740845 13267 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:21:56.741180 13267 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms:/tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:21:56.741254 13267 mini_ranger_kms.cc:240] Using host: 127.12.244.212
I20260926 02:21:56.767390 13267 mini_ranger_kms.cc:276] Ranger KMS PID: 24590
I20260926 02:21:56.767632 13267 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.244.212:44285
Sep 26, 2026 2:21:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:21:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:21:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:21:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:21:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:21:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 02:21:59.132020 13267 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.018s user 0.120s sys 0.349s
I20260926 02:21:59.132306 13267 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:21:59.132515 13267 mini_ranger_kms.cc:401] 127.12.244.212:44285/kms/v1/keys
Sep 26 02:21:59 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790389319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.244.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 02:21:59 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.244.212@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
Sep 26 02:22:00 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389319, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.244.212@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
Sep 26 02:22:02 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:22:02.467448 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46061
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:46061
--ranger_config_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:44285/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:22:02.835748 24640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:02.836143 24640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:02.836316 24640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:02.843958 24640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:22:02.844113 24640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:02.844224 24640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:02.844319 24640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:22:02.844410 24640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:22:02.855614 24640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.244.212:44285/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:46061
--ranger_config_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:02.857057 24640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:02.858445 24640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:02.872186 24645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:02.872308 24648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:02.872903 24640 server_base.cc:1034] running on GCE node
W20260926 02:22:02.872260 24646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:02.874050 24640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:02.875624 24640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:02.876850 24640 hybrid_clock.cc:648] HybridClock initialized: now 1790389322876783 us; error 62 us; skew 500 ppm
I20260926 02:22:02.877280 24640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:02 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:02.898434 24640 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:22:02.900327 24640 webserver.cc:460] Webserver started at http://127.12.244.254:40441/ using document root <none> and password file <none>
I20260926 02:22:02.901055 24640 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:02.901186 24640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:02.901571 24640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:02 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389322, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:02.963182 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "42db878812f4473286960c62f66658f8"
format_stamp: "Formatted at 2026-09-26 02:22:02 on dist-test-slave-qfmw"
server_key: "6871dcdc3c61418842771fd9302e0cc7"
server_key_iv: "123d5147bbebe1246933a9bd539fb224"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:02.964242 24640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "42db878812f4473286960c62f66658f8"
format_stamp: "Formatted at 2026-09-26 02:22:02 on dist-test-slave-qfmw"
server_key: "6871dcdc3c61418842771fd9302e0cc7"
server_key_iv: "123d5147bbebe1246933a9bd539fb224"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:02.970361 24640 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260926 02:22:02.991976 24655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:02.993844 24640 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:22:02.994174 24640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "42db878812f4473286960c62f66658f8"
format_stamp: "Formatted at 2026-09-26 02:22:02 on dist-test-slave-qfmw"
server_key: "6871dcdc3c61418842771fd9302e0cc7"
server_key_iv: "123d5147bbebe1246933a9bd539fb224"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:02.994470 24640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:03.020381 24640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:03.025236 24640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:03.025568 24640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:03.046995 24640 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:46061
I20260926 02:22:03.047050 24706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:46061 every 8 connection(s)
I20260926 02:22:03.048538 24640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:22:03.053866 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:22:03.058890 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 24640
I20260926 02:22:03.059187 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:22:03.060072 13267 external_mini_cluster.cc:1428] Setting key 425bf6f6164b6ba2685d35f31a0426ed
I20260926 02:22:03.068079 24707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8: Bootstrap starting.
I20260926 02:22:03.073035 24707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8: Neither blocks nor log segments found. Creating new log.
I20260926 02:22:03.074424 24707 log.cc:824] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:22:03.078516 24707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8: No bootstrap required, opened a new log
Sep 26 02:22:03 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:22:03.089511 24707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db878812f4473286960c62f66658f8" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46061 } }
I20260926 02:22:03.090000 24707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:22:03.090158 24707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42db878812f4473286960c62f66658f8, State: Initialized, Role: FOLLOWER
I20260926 02:22:03.090198 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:03.062576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.12.244.254:46061)
0926 02:22:03.063604 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:03.063621 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:03.063676 (+ 55us) server_negotiation.cc:407] Connection header received
0926 02:22:03.066222 (+ 2546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:03.066253 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:03.066751 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:03.067248 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:03.069144 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:03.071054 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:03.073352 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:03.074252 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:03.074411 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:03.080521 (+ 6110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:03.080532 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:03.080557 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:03.080611 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:03.084170 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:03.085747 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:03.085756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:03.085764 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:03.085982 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:03.086538 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:03.086543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:03.086545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:03.087391 (+ 846us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:03.087634 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:03.088309 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 02:22:03.088810 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":507,"threads_started":1}
I20260926 02:22:03.091146 24707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db878812f4473286960c62f66658f8" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46061 } }
I20260926 02:22:03.091468 24707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:22:03.091641 24707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:22:03.091845 24707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:22:03.094741 24707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db878812f4473286960c62f66658f8" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46061 } }
I20260926 02:22:03.095378 24707 leader_election.cc:302] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42db878812f4473286960c62f66658f8; no voters:
I20260926 02:22:03.096051 24707 leader_election.cc:288] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:22:03.096138 24712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:22:03.097790 24712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [term 1 LEADER]: Becoming Leader. State: Replica: 42db878812f4473286960c62f66658f8, State: Running, Role: LEADER
I20260926 02:22:03.098475 24707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:22:03.098431 24712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db878812f4473286960c62f66658f8" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46061 } }
I20260926 02:22:03.102700 24713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42db878812f4473286960c62f66658f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db878812f4473286960c62f66658f8" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46061 } } }
I20260926 02:22:03.103223 24713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:22:03.102664 24714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42db878812f4473286960c62f66658f8. Latest consensus state: current_term: 1 leader_uuid: "42db878812f4473286960c62f66658f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42db878812f4473286960c62f66658f8" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 46061 } } }
I20260926 02:22:03.103650 24714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8 [sys.catalog]: This master's current role is: LEADER
I20260926 02:22:03.107237 24721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:22:03.114209 24721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:22:03.115031 24707 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:22:03.132766 24721 catalog_manager.cc:1347] Generated new cluster ID: bbcfc9eae1b346c982f2f30a9a6eed94
I20260926 02:22:03.133057 24721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:22:03.159461 24721 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:22:03.162320 24721 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:22:03.184783 24721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8: Generated new TSK 0
I20260926 02:22:03.186311 24721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:22:03.963910 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:44285/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46061
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:22:04 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790389324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:22:04.435556 24752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:04.435999 24752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:04.436200 24752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:04.448271 24752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:04.448441 24752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:04.448720 24752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:22:04.465845 24752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.244.212:44285/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:04.467960 24752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:04.469830 24752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:04.496050 24762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:04.500839 24763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:04.506477 24752 server_base.cc:1034] running on GCE node
W20260926 02:22:04.507077 24765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:04.508057 24752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:04.509362 24752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:04.510625 24752 hybrid_clock.cc:648] HybridClock initialized: now 1790389324510546 us; error 77 us; skew 500 ppm
I20260926 02:22:04.510998 24752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:04 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:04.514642 24752 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:22:04.516594 24752 webserver.cc:460] Webserver started at http://127.12.244.193:45741/ using document root <none> and password file <none>
I20260926 02:22:04.517400 24752 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:04.517558 24752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:04.517889 24752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:04 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:04.536631 24752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "22a8c0e4c5e84e1289511ca835a674a7"
format_stamp: "Formatted at 2026-09-26 02:22:04 on dist-test-slave-qfmw"
server_key: "6960877fd2c55d64d969362252e4dfed"
server_key_iv: "2a307836930642a7dffc097d3f42153a"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:04.537612 24752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "22a8c0e4c5e84e1289511ca835a674a7"
format_stamp: "Formatted at 2026-09-26 02:22:04 on dist-test-slave-qfmw"
server_key: "6960877fd2c55d64d969362252e4dfed"
server_key_iv: "2a307836930642a7dffc097d3f42153a"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:04.543818 24752 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 02:22:04.583504 24772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:04.585899 24752 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.013s sys 0.007s
I20260926 02:22:04.586254 24752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "22a8c0e4c5e84e1289511ca835a674a7"
format_stamp: "Formatted at 2026-09-26 02:22:04 on dist-test-slave-qfmw"
server_key: "6960877fd2c55d64d969362252e4dfed"
server_key_iv: "2a307836930642a7dffc097d3f42153a"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:04.586742 24752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:04.667989 24752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:04.674378 24752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:04.674813 24752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:04.676412 24752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:22:04.684360 24752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:22:04.684520 24752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:04.684686 24752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:22:04.684772 24752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:22:04 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:04.820747 24752 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:42597
I20260926 02:22:04.820814 24886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:42597 every 8 connection(s)
I20260926 02:22:04.822531 24752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:22:04.823964 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 24752
I20260926 02:22:04.824177 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:22:04.825017 13267 external_mini_cluster.cc:1428] Setting key 434aad55f8ef774ef3431c0878cef5c7
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Sep 26 02:22:04 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:22:04.938915 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:04.858056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46061 (local address 127.12.244.193:32953)
0926 02:22:04.896507 (+ 38451us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:04.896577 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:04.899644 (+ 3067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:04.902393 (+ 2749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:04.902407 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:04.910331 (+ 7924us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:22:04.911425 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:04.911461 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:04.913198 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:04.913204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:04.915564 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:04.915584 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:04.917502 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:04.917510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:04.917865 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:04.923852 (+ 5987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:22:04.923895 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:22:04.928185 (+ 4290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:22:04.932917 (+ 4732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:04.932929 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:04.932952 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:04.933671 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:04.934205 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:04.934210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:04.934214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:04.935027 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:04.935831 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:22:04.935841 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:22:04.936385 (+ 544us) client_negotiation.cc:769] Sending connection context
0926 02:22:04.936739 (+ 354us) client_negotiation.cc:240] Negotiation successful
0926 02:22:04.937363 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38033,"thread_start_us":40956,"threads_started":1}
I20260926 02:22:04.940448 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:04.858801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:32953 (local address 127.12.244.254:46061)
0926 02:22:04.891492 (+ 32691us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:04.891500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:04.896727 (+ 5227us) server_negotiation.cc:407] Connection header received
0926 02:22:04.900021 (+ 3294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:04.900027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:04.900142 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:04.900300 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:04.911725 (+ 11425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:04.912909 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:04.915782 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:04.916928 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:04.917077 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:04.928612 (+ 11535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:04.928621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:04.928627 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:04.928679 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:04.932661 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:04.933852 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:04.933858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:04.933862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:04.934045 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:04.935201 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:04.935207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:04.935211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:04.935589 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:04.935746 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:04.939764 (+ 4018us) server_negotiation.cc:299] Negotiation successful
0926 02:22:04.940051 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32530,"thread_start_us":32710,"threads_started":1}
I20260926 02:22:04.942270 24887 heartbeater.cc:347] Connected to a master server at 127.12.244.254:46061
I20260926 02:22:04.942641 24887 heartbeater.cc:464] Registering TS with master...
I20260926 02:22:04.943573 24887 heartbeater.cc:511] Master 127.12.244.254:46061 requested a full tablet report, sending...
I20260926 02:22:04.946400 24672 ts_manager.cc:194] Registered new tserver with Master: 22a8c0e4c5e84e1289511ca835a674a7 (127.12.244.193:42597)
I20260926 02:22:04.948587 24672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:32953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
W20260926 02:22:04.959398 13267 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.244.194: real 0.131s user 0.000s sys 0.017s
I20260926 02:22:04.960364 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:44285/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46061
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:22:05 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389324, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
W20260926 02:22:05.514314 24893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:05.514698 24893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:05.514863 24893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:05.523608 24893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:05.523798 24893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:05.523969 24893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:22:05.534886 24893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.244.212:44285/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:05.536475 24893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:05.537782 24893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:05.553310 24907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:05.553268 24908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:05.553228 24910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:05.555058 24893 server_base.cc:1034] running on GCE node
I20260926 02:22:05.555855 24893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:05.556979 24893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:05.558228 24893 hybrid_clock.cc:648] HybridClock initialized: now 1790389325558176 us; error 40 us; skew 500 ppm
I20260926 02:22:05.558573 24893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:05 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:05.562057 24893 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:22:05.563751 24893 webserver.cc:460] Webserver started at http://127.12.244.194:34737/ using document root <none> and password file <none>
I20260926 02:22:05.564479 24893 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:05.564596 24893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:05.564877 24893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:05 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:05.584050 24893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "ff028a6183f244fd99484ee0caf041c1"
format_stamp: "Formatted at 2026-09-26 02:22:05 on dist-test-slave-qfmw"
server_key: "b4f27f87818ec265638678bf5fa04cc4"
server_key_iv: "5added1b458f378e06f2ad70d5dca6f4"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:05.585227 24893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "ff028a6183f244fd99484ee0caf041c1"
format_stamp: "Formatted at 2026-09-26 02:22:05 on dist-test-slave-qfmw"
server_key: "b4f27f87818ec265638678bf5fa04cc4"
server_key_iv: "5added1b458f378e06f2ad70d5dca6f4"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:05.593672 24893 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260926 02:22:05.611960 24917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:05.613869 24893 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:22:05.614087 24893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "ff028a6183f244fd99484ee0caf041c1"
format_stamp: "Formatted at 2026-09-26 02:22:05 on dist-test-slave-qfmw"
server_key: "b4f27f87818ec265638678bf5fa04cc4"
server_key_iv: "5added1b458f378e06f2ad70d5dca6f4"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:05.614363 24893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:05.629189 24893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:05.633250 24893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:05.633581 24893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:05.634608 24893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:22:05.636132 24893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:22:05.636216 24893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:05.636373 24893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:22:05.636451 24893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:05.673767 24893 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:37521
I20260926 02:22:05.673815 25030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:37521 every 8 connection(s)
I20260926 02:22:05.675186 24893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:22:05.682034 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 24893
I20260926 02:22:05.682225 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:22:05.682804 13267 external_mini_cluster.cc:1428] Setting key 9ed855adaba4e84f49ac5295758a66ee
Sep 26 02:22:05 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389325, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:22:05.698076 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:05.677811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:57805 (local address 127.12.244.254:46061)
0926 02:22:05.678250 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:05.678254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:05.679294 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 02:22:05.681521 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:05.681526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:05.681612 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:05.681728 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:05.683896 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:05.684827 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:05.686486 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:05.686926 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:05.687091 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:05.691145 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:05.691159 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:05.691163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:05.691204 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:05.694592 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:05.695573 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:05.695578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:05.695581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:05.695708 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:05.696169 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:05.696174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:05.696176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:05.696471 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:05.696635 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:05.697444 (+ 809us) server_negotiation.cc:299] Negotiation successful
0926 02:22:05.697696 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":209,"threads_started":1}
I20260926 02:22:05.699348 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:05.678416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46061 (local address 127.12.244.194:57805)
0926 02:22:05.679081 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:05.679148 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:05.681193 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:05.682082 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:05.682094 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:05.682765 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:22:05.683684 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:05.683715 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:05.684995 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:05.685005 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:05.686324 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:05.686337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:05.687227 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:05.687235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:05.687486 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:05.688309 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:22:05.688334 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:22:05.690822 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:22:05.694789 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:05.694802 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:05.694820 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:05.695411 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:05.695827 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:05.695832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:05.695838 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:05.696029 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:05.696622 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:22:05.696631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:22:05.697006 (+ 375us) client_negotiation.cc:769] Sending connection context
0926 02:22:05.697319 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 02:22:05.697878 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":265,"threads_started":1}
I20260926 02:22:05.701129 25031 heartbeater.cc:347] Connected to a master server at 127.12.244.254:46061
I20260926 02:22:05.701542 25031 heartbeater.cc:464] Registering TS with master...
I20260926 02:22:05.702564 25031 heartbeater.cc:511] Master 127.12.244.254:46061 requested a full tablet report, sending...
I20260926 02:22:05.704473 24672 ts_manager.cc:194] Registered new tserver with Master: ff028a6183f244fd99484ee0caf041c1 (127.12.244.194:37521)
I20260926 02:22:05.705338 24672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:57805
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:22:05.738976 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:44285/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46061
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:40683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:22:05.953429 24887 heartbeater.cc:503] Master 127.12.244.254:46061 was elected leader, sending a full tablet report...
W20260926 02:22:06.106763 25037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:06.107092 25037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:06.107286 25037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:06.114665 25037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:06.114787 25037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:06.114988 25037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:22:06.126282 25037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:40683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.244.212:44285/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:46061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:06.127820 25037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:06.129194 25037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:06.143653 25042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:06.143682 25043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:06.143682 25045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:06.150122 25037 server_base.cc:1034] running on GCE node
I20260926 02:22:06.151173 25037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:06.152537 25037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:06.153927 25037 hybrid_clock.cc:648] HybridClock initialized: now 1790389326153855 us; error 84 us; skew 500 ppm
I20260926 02:22:06.154443 25037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:06 dist-test-slave-qfmw krb5kdc[24193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:06.158897 25037 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:22:06.160784 25037 webserver.cc:460] Webserver started at http://127.12.244.195:40429/ using document root <none> and password file <none>
I20260926 02:22:06.161597 25037 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:06.161765 25037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:06.162096 25037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:06 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:06.176929 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "4e7c250a92644fcc84d606e320818da5"
format_stamp: "Formatted at 2026-09-26 02:22:06 on dist-test-slave-qfmw"
server_key: "254753e999923a47309e4a60098b542c"
server_key_iv: "7c9fd1a3e80af249c82a3e58eee52f61"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:06.177843 25037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "4e7c250a92644fcc84d606e320818da5"
format_stamp: "Formatted at 2026-09-26 02:22:06 on dist-test-slave-qfmw"
server_key: "254753e999923a47309e4a60098b542c"
server_key_iv: "7c9fd1a3e80af249c82a3e58eee52f61"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:06.183902 25037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 02:22:06.199908 25052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:06.201717 25037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 02:22:06.201924 25037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "4e7c250a92644fcc84d606e320818da5"
format_stamp: "Formatted at 2026-09-26 02:22:06 on dist-test-slave-qfmw"
server_key: "254753e999923a47309e4a60098b542c"
server_key_iv: "7c9fd1a3e80af249c82a3e58eee52f61"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:06.202188 25037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:06.229830 25037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:06.233898 25037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:06.234254 25037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:06.235404 25037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:22:06.236902 25037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:22:06.237010 25037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:06.237176 25037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:22:06.237251 25037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:06.276954 25037 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:40075
I20260926 02:22:06.277068 25165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:40075 every 8 connection(s)
I20260926 02:22:06.279335 25037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:22:06.285935 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 25037
I20260926 02:22:06.286129 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:22:06.286667 13267 external_mini_cluster.cc:1428] Setting key 0f6d79c3b3b8106d1ab4604a23a17e06
Sep 26 02:22:06 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389326, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:22:06.303215 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.282322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:58251 (local address 127.12.244.254:46061)
0926 02:22:06.282808 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.282815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.283974 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 02:22:06.286034 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.286041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.286177 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.286306 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:06.288154 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.289100 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.291291 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.291788 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.291896 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.296109 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:06.296114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:06.296117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:06.296156 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:06.299612 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.300554 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.300559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.300561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.300743 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.301232 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.301239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.301246 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.301623 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:06.301786 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.302575 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.302811 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":212,"threads_started":1}
I20260926 02:22:06.304339 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.282969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:46061 (local address 127.12.244.195:58251)
0926 02:22:06.283800 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:06.283840 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:06.285703 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:06.286579 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:06.286589 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:06.287178 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:22:06.287944 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.287963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.289358 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.289363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.291060 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.291075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.291997 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.292002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.292771 (+ 769us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.293511 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:22:06.293541 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:22:06.295824 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:22:06.299809 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:06.299818 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:06.299836 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:06.300395 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:06.300873 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:06.300878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:06.300882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:06.301095 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:06.301781 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:22:06.301790 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:22:06.302198 (+ 408us) client_negotiation.cc:769] Sending connection context
0926 02:22:06.302486 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 02:22:06.303069 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":295,"threads_started":1}
I20260926 02:22:06.306007 25166 heartbeater.cc:347] Connected to a master server at 127.12.244.254:46061
I20260926 02:22:06.306378 25166 heartbeater.cc:464] Registering TS with master...
I20260926 02:22:06.307413 25166 heartbeater.cc:511] Master 127.12.244.254:46061 requested a full tablet report, sending...
I20260926 02:22:06.309051 24672 ts_manager.cc:194] Registered new tserver with Master: 4e7c250a92644fcc84d606e320818da5 (127.12.244.195:40075)
I20260926 02:22:06.309908 24672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:58251
I20260926 02:22:06.315925 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:22:06.335458 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.321029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.12.244.254:46061)
0926 02:22:06.321320 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.321325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.322093 (+ 768us) server_negotiation.cc:407] Connection header received
0926 02:22:06.322167 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.322171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.322260 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.322362 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:06.323911 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.324707 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.326141 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.326636 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.326757 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.329498 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:06.329504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:06.329508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:06.329548 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:06.332733 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.333499 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.333504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.333506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.333666 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.334028 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.334031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.334033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.334301 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:06.334483 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.334921 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.335145 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 02:22:06.340421 24672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:22:06.348361 24672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:22:06.376775 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.362731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.12.244.194:37521)
0926 02:22:06.364667 (+ 1936us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.364681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.366928 (+ 2247us) server_negotiation.cc:407] Connection header received
0926 02:22:06.367049 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.367058 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.367389 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.367598 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:22:06.370074 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.371665 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.373882 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.375160 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.375346 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.375561 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.375802 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.376403 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1708,"thread_start_us":1575,"threads_started":1}
I20260926 02:22:06.376835 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.362422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:37521 (local address 127.0.0.1:43856)
0926 02:22:06.366618 (+ 4196us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:06.366642 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:06.366784 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:06.367846 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:06.367850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:06.367871 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:22:06.369843 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.369859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.371848 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.371856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.373696 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.373712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.375352 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.375359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.375545 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.375600 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 02:22:06.376249 (+ 649us) client_negotiation.cc:240] Negotiation successful
0926 02:22:06.376447 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4060,"mutex_wait_us":64,"thread_start_us":3933,"threads_started":1}
I20260926 02:22:06.377342 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.362020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:40075 (local address 127.0.0.1:54776)
0926 02:22:06.363580 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:06.363639 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:06.363852 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:06.367653 (+ 3801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:06.367663 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:06.367693 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:22:06.369090 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.369106 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.371497 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.371503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.374533 (+ 3030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.374546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.376500 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.376507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.376685 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.376725 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:22:06.376792 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 02:22:06.376936 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":63,"thread_start_us":1007,"threads_started":1}
I20260926 02:22:06.377895 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.362226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.12.244.195:40075)
0926 02:22:06.367073 (+ 4847us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.367082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.367138 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:22:06.367220 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.367226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.367401 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.368671 (+ 1270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:22:06.369573 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.371345 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.374719 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.376271 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.377157 (+ 886us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.377354 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.377431 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.377594 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4667,"thread_start_us":4515,"threads_started":1}
I20260926 02:22:06.378564 25184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.364457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.12.244.193:42597)
0926 02:22:06.369314 (+ 4857us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.369324 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.369877 (+ 553us) server_negotiation.cc:407] Connection header received
0926 02:22:06.369997 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.370006 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.370290 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.370509 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:22:06.371119 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.372695 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.375183 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.376678 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.376872 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.377376 (+ 504us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.377967 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.378238 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4665,"thread_start_us":4246,"threads_started":1}
I20260926 02:22:06.381044 24966 tablet_service.cc:1467] Processing CreateTablet for tablet 47d15fcae0e24ee38578f6ccec7b0fc3 (DEFAULT_TABLE table=test-table [id=d127d216855b47cd8cd73a9da5de56ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:22:06.381238 25101 tablet_service.cc:1467] Processing CreateTablet for tablet 47d15fcae0e24ee38578f6ccec7b0fc3 (DEFAULT_TABLE table=test-table [id=d127d216855b47cd8cd73a9da5de56ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:22:06.381843 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.364131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:42597 (local address 127.0.0.1:47442)
0926 02:22:06.369573 (+ 5442us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:06.369600 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:06.369735 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:06.370492 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:06.370497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:06.370523 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:22:06.370926 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.370934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.372872 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.372878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.374975 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.374986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.377526 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.377531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.377757 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.377800 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 02:22:06.377890 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:22:06.378019 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5311,"mutex_wait_us":31,"thread_start_us":5178,"threads_started":1}
I20260926 02:22:06.382494 25101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d15fcae0e24ee38578f6ccec7b0fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:22:06.382499 24966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d15fcae0e24ee38578f6ccec7b0fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:22:06.385978 24821 tablet_service.cc:1467] Processing CreateTablet for tablet 47d15fcae0e24ee38578f6ccec7b0fc3 (DEFAULT_TABLE table=test-table [id=d127d216855b47cd8cd73a9da5de56ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:22:06.387714 24821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d15fcae0e24ee38578f6ccec7b0fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:22:06.395213 25185 tablet_bootstrap.cc:492] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Bootstrap starting.
I20260926 02:22:06.398533 25186 tablet_bootstrap.cc:492] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Bootstrap starting.
I20260926 02:22:06.399562 25185 tablet_bootstrap.cc:654] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Neither blocks nor log segments found. Creating new log.
I20260926 02:22:06.401070 25185 log.cc:824] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:22:06.402850 25186 tablet_bootstrap.cc:654] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Neither blocks nor log segments found. Creating new log.
I20260926 02:22:06.404527 25186 log.cc:824] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:22:06.404963 25188 tablet_bootstrap.cc:492] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Bootstrap starting.
I20260926 02:22:06.409291 25185 tablet_bootstrap.cc:492] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: No bootstrap required, opened a new log
I20260926 02:22:06.409628 25188 tablet_bootstrap.cc:654] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Neither blocks nor log segments found. Creating new log.
I20260926 02:22:06.409677 25185 ts_tablet_manager.cc:1397] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260926 02:22:06.409778 25186 tablet_bootstrap.cc:492] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: No bootstrap required, opened a new log
I20260926 02:22:06.410068 25186 ts_tablet_manager.cc:1397] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 02:22:06.411129 25188 log.cc:824] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:22:06.415716 25188 tablet_bootstrap.cc:492] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: No bootstrap required, opened a new log
I20260926 02:22:06.416047 25188 ts_tablet_manager.cc:1397] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260926 02:22:06.419446 25186 raft_consensus.cc:348] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.419873 25186 raft_consensus.cc:374] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:22:06.419960 25186 raft_consensus.cc:729] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7c250a92644fcc84d606e320818da5, State: Initialized, Role: FOLLOWER
I20260926 02:22:06.420717 25186 consensus_queue.cc:260] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.420667 25185 raft_consensus.cc:348] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.421092 25185 raft_consensus.cc:374] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:22:06.421183 25185 raft_consensus.cc:729] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff028a6183f244fd99484ee0caf041c1, State: Initialized, Role: FOLLOWER
I20260926 02:22:06.421965 25166 heartbeater.cc:503] Master 127.12.244.254:46061 was elected leader, sending a full tablet report...
I20260926 02:22:06.422890 25186 ts_tablet_manager.cc:1428] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260926 02:22:06.423882 25185 consensus_queue.cc:260] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.425107 25031 heartbeater.cc:503] Master 127.12.244.254:46061 was elected leader, sending a full tablet report...
I20260926 02:22:06.427831 25188 raft_consensus.cc:348] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.428329 25188 raft_consensus.cc:374] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:22:06.428429 25188 raft_consensus.cc:729] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a8c0e4c5e84e1289511ca835a674a7, State: Initialized, Role: FOLLOWER
I20260926 02:22:06.428992 25185 ts_tablet_manager.cc:1428] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Time spent starting tablet: real 0.019s user 0.010s sys 0.005s
I20260926 02:22:06.429090 25188 consensus_queue.cc:260] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.431077 25188 ts_tablet_manager.cc:1428] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260926 02:22:06.453058 25191 raft_consensus.cc:482] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:22:06.453384 25191 raft_consensus.cc:504] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.455228 25191 leader_election.cc:288] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22a8c0e4c5e84e1289511ca835a674a7 (127.12.244.193:42597), ff028a6183f244fd99484ee0caf041c1 (127.12.244.194:37521)
I20260926 02:22:06.461612 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.455565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:42597 (local address 127.12.244.195:56695)
0926 02:22:06.455801 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:06.455824 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:06.455959 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:06.456404 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:06.456419 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:06.456451 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:22:06.456741 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.456756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.457905 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.457916 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.459723 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.459743 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.460968 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.460976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.461142 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.461167 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:22:06.461302 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 02:22:06.461406 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 02:22:06.461742 25184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.455807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:56695 (local address 127.12.244.193:42597)
0926 02:22:06.456020 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.456026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.456051 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:22:06.456138 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.456146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.456239 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.456361 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:22:06.456896 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.457752 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.459899 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.460807 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.460937 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.461039 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.461340 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.461479 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:22:06.462874 24841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d15fcae0e24ee38578f6ccec7b0fc3" candidate_uuid: "4e7c250a92644fcc84d606e320818da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a8c0e4c5e84e1289511ca835a674a7" is_pre_election: true
I20260926 02:22:06.463310 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.455848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:38849 (local address 127.12.244.194:37521)
0926 02:22:06.456065 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.456072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.456790 (+ 718us) server_negotiation.cc:407] Connection header received
0926 02:22:06.456920 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.456926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.457006 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.457106 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:22:06.458332 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.459020 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.461068 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.462226 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.462369 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.462460 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.462891 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.463018 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:22:06.463586 24841 raft_consensus.cc:2454] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7c250a92644fcc84d606e320818da5 in term 0.
I20260926 02:22:06.463827 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.455639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:37521 (local address 127.12.244.195:38849)
0926 02:22:06.456670 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:06.456692 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:06.456810 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:06.457172 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:06.457175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:06.457197 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:22:06.458135 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.458147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.459183 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.459188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.460857 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:06.460897 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.462450 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.462456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:06.462702 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.462743 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:22:06.462848 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:22:06.462983 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":267,"thread_start_us":389,"threads_started":1}
I20260926 02:22:06.464326 25055 leader_election.cc:302] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22a8c0e4c5e84e1289511ca835a674a7, 4e7c250a92644fcc84d606e320818da5; no voters:
I20260926 02:22:06.464802 25191 raft_consensus.cc:2781] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:22:06.464967 25191 raft_consensus.cc:482] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:22:06.465082 25191 raft_consensus.cc:3037] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:22:06.464963 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d15fcae0e24ee38578f6ccec7b0fc3" candidate_uuid: "4e7c250a92644fcc84d606e320818da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff028a6183f244fd99484ee0caf041c1" is_pre_election: true
I20260926 02:22:06.465489 24986 raft_consensus.cc:2454] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7c250a92644fcc84d606e320818da5 in term 0.
I20260926 02:22:06.467602 25191 raft_consensus.cc:504] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.468210 25191 leader_election.cc:288] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [CANDIDATE]: Term 1 election: Requested vote from peers 22a8c0e4c5e84e1289511ca835a674a7 (127.12.244.193:42597), ff028a6183f244fd99484ee0caf041c1 (127.12.244.194:37521)
I20260926 02:22:06.468780 24986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d15fcae0e24ee38578f6ccec7b0fc3" candidate_uuid: "4e7c250a92644fcc84d606e320818da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff028a6183f244fd99484ee0caf041c1"
I20260926 02:22:06.468781 24841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d15fcae0e24ee38578f6ccec7b0fc3" candidate_uuid: "4e7c250a92644fcc84d606e320818da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22a8c0e4c5e84e1289511ca835a674a7"
I20260926 02:22:06.469004 24986 raft_consensus.cc:3037] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:22:06.469000 24841 raft_consensus.cc:3037] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:22:06.472823 24986 raft_consensus.cc:2454] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7c250a92644fcc84d606e320818da5 in term 1.
I20260926 02:22:06.472823 24841 raft_consensus.cc:2454] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7c250a92644fcc84d606e320818da5 in term 1.
I20260926 02:22:06.473384 25054 leader_election.cc:302] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7c250a92644fcc84d606e320818da5, ff028a6183f244fd99484ee0caf041c1; no voters:
I20260926 02:22:06.473845 25191 raft_consensus.cc:2781] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:22:06.474261 25191 raft_consensus.cc:686] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e7c250a92644fcc84d606e320818da5, State: Running, Role: LEADER
I20260926 02:22:06.474751 25191 consensus_queue.cc:237] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } }
I20260926 02:22:06.479724 24672 catalog_manager.cc:5495] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e7c250a92644fcc84d606e320818da5 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "4e7c250a92644fcc84d606e320818da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7c250a92644fcc84d606e320818da5" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 40075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 } health_report { overall_health: UNKNOWN } } }
I20260926 02:22:06.501358 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.495760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.12.244.195:40075)
0926 02:22:06.495980 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.495986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.496008 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:22:06.496144 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.496148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.496224 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.496388 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:22:06.496824 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.497538 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.499087 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.499532 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.499696 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.500235 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:22:06.500784 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:22:06.500946 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.501002 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.501083 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:22:06.510836 24841 raft_consensus.cc:1261] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 1 FOLLOWER]: Refusing update from remote peer 4e7c250a92644fcc84d606e320818da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:22:06.510829 24986 raft_consensus.cc:1261] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 1 FOLLOWER]: Refusing update from remote peer 4e7c250a92644fcc84d606e320818da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:22:06.512017 25195 consensus_queue.cc:1035] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22a8c0e4c5e84e1289511ca835a674a7" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:22:06.512413 25191 consensus_queue.cc:1035] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff028a6183f244fd99484ee0caf041c1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:22:06.524878 25198 mvcc.cc:205] Tried to move back new op lower bound from 7333434681377804288 to 7333434681244876800. Current Snapshot: MvccSnapshot[applied={T|T < 7333434681377804288}]
I20260926 02:22:06.537814 25200 mvcc.cc:205] Tried to move back new op lower bound from 7333434681377804288 to 7333434681244876800. Current Snapshot: MvccSnapshot[applied={T|T < 7333434681377804288}]
I20260926 02:22:06.546751 24672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:22:06.547014 24672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:22:06.550765 24672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42db878812f4473286960c62f66658f8: Sending DeleteTablet for 3 replicas of tablet 47d15fcae0e24ee38578f6ccec7b0fc3
I20260926 02:22:06.552107 24966 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d15fcae0e24ee38578f6ccec7b0fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:22:06 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:43856
I20260926 02:22:06.552192 24821 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d15fcae0e24ee38578f6ccec7b0fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:22:06 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:47442
I20260926 02:22:06.552497 25101 tablet_service.cc:1514] Processing DeleteTablet for tablet 47d15fcae0e24ee38578f6ccec7b0fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:22:06 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:54776
I20260926 02:22:06.554840 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 24752
I20260926 02:22:06.567649 25208 tablet_replica.cc:330] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: stopping tablet replica
I20260926 02:22:06.568324 25208 raft_consensus.cc:2229] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:22:06.569022 25208 raft_consensus.cc:2258] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:22:06.572001 25208 ts_tablet_manager.cc:1905] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:22:06.582320 25208 ts_tablet_manager.cc:1918] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:22:06.582572 25208 log.cc:1196] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/47d15fcae0e24ee38578f6ccec7b0fc3
I20260926 02:22:06.583761 25208 ts_tablet_manager.cc:1939] T 47d15fcae0e24ee38578f6ccec7b0fc3 P ff028a6183f244fd99484ee0caf041c1: Deleting consensus metadata
I20260926 02:22:06.586297 24657 catalog_manager.cc:4841] TS ff028a6183f244fd99484ee0caf041c1 (127.12.244.194:37521): tablet 47d15fcae0e24ee38578f6ccec7b0fc3 (table test-table [id=d127d216855b47cd8cd73a9da5de56ca]) successfully deleted
I20260926 02:22:06.587698 25209 tablet_replica.cc:330] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: stopping tablet replica
I20260926 02:22:06.587713 25210 tablet_replica.cc:330] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: stopping tablet replica
I20260926 02:22:06.588183 25210 raft_consensus.cc:2229] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:22:06.588478 25209 raft_consensus.cc:2229] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:22:06.588883 25210 raft_consensus.cc:2258] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:22:06.589123 25209 raft_consensus.cc:2258] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:22:06.591373 25209 ts_tablet_manager.cc:1905] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:22:06.591383 25210 ts_tablet_manager.cc:1905] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:22:06 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:22:06.599524 25209 ts_tablet_manager.cc:1918] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:22:06.599697 25210 ts_tablet_manager.cc:1918] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:22:06.600055 25210 log.cc:1196] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/47d15fcae0e24ee38578f6ccec7b0fc3
I20260926 02:22:06.600049 25209 log.cc:1196] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/47d15fcae0e24ee38578f6ccec7b0fc3
I20260926 02:22:06.601125 25210 ts_tablet_manager.cc:1939] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 4e7c250a92644fcc84d606e320818da5: Deleting consensus metadata
I20260926 02:22:06.601359 25209 ts_tablet_manager.cc:1939] T 47d15fcae0e24ee38578f6ccec7b0fc3 P 22a8c0e4c5e84e1289511ca835a674a7: Deleting consensus metadata
I20260926 02:22:06.602862 25184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.556603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.12.244.193:42597)
0926 02:22:06.556796 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.556804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.580006 (+ 23202us) server_negotiation.cc:407] Connection header received
0926 02:22:06.580583 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.580589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.580689 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.580770 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:06.582874 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.584078 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.587571 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.588276 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.588383 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.594355 (+ 5972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:06.594374 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:06.594394 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:06.594447 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:06.598743 (+ 4296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.599954 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.599967 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.599970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.600211 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.600680 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.600686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.600692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.601216 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:06.601522 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.602137 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.602416 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:22:06.603118 24658 catalog_manager.cc:4841] TS 4e7c250a92644fcc84d606e320818da5 (127.12.244.195:40075): tablet 47d15fcae0e24ee38578f6ccec7b0fc3 (table test-table [id=d127d216855b47cd8cd73a9da5de56ca]) successfully deleted
I20260926 02:22:06.603850 24658 catalog_manager.cc:4841] TS 22a8c0e4c5e84e1289511ca835a674a7 (127.12.244.193:42597): tablet 47d15fcae0e24ee38578f6ccec7b0fc3 (table test-table [id=d127d216855b47cd8cd73a9da5de56ca]) successfully deleted
I20260926 02:22:06.604445 24882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:06.695983 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 24752
I20260926 02:22:06.718920 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 24893
Sep 26 02:22:06 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:22:06.735976 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.719985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.12.244.194:37521)
0926 02:22:06.720179 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.720186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.720208 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:22:06.720374 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.720381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.720460 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.720587 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:06.721972 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.723173 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.724563 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.725191 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.725319 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.729951 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:06.729966 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:06.729984 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:06.730024 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:06.733104 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.733968 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.733976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.733978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.734105 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.734500 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.734503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.734505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.734879 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:06.735266 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.735505 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.735667 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:22:06.736651 25026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:06.843554 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 24893
I20260926 02:22:06.860627 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 25037
Sep 26 02:22:06 dist-test-slave-qfmw krb5kdc[24193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:22:06.881235 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:06.861507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54792 (local address 127.12.244.195:40075)
0926 02:22:06.861675 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:06.861680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:06.861720 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:22:06.861972 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:06.861977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:06.862114 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:06.862220 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:06.864122 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.864923 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.866892 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:06.867418 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:06.867523 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:06.873288 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:06.873302 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:06.873322 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:06.873375 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:06.877577 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.878768 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.878784 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.878787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.878981 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:06.879446 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:06.879457 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:06.879461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:06.879967 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:06.880321 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:06.880565 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 02:22:06.880818 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:22:06.882000 25161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:07.006832 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 25037
I20260926 02:22:07.027428 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 24640
I20260926 02:22:07.028746 24702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:07.143352 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 24640
I20260926 02:22:07.165700 13267 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:22:07.501605 13267 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:22:07.501871 13267 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:22:07.848086 13267 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:22:07.848 UTC [24224] LOG: received smart shutdown request
2026-09-26 02:22:07.851 UTC [24224] LOG: background worker "logical replication launcher" (PID 24233) exited with exit code 1
2026-09-26 02:22:07.852 UTC [24228] LOG: shutting down
2026-09-26 02:22:08.047 UTC [24224] LOG: database system is shut down
2026-09-26T02:22:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53220 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:22:08 dist-test-slave-qfmw krb5kdc[25224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:22:08 dist-test-slave-qfmw krb5kdc[25224](info): set up 2 sockets
Sep 26 02:22:08 dist-test-slave-qfmw krb5kdc[25224](info): commencing operation
krb5kdc: starting...
W20260926 02:22:09.313998 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:22:09 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:22:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/rangeradmin_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/rangeradmin_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/rangerlookup_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/rangerlookup_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
I20260926 02:22:09.669776 13267 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3VVuru/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/postgres -l logfile start
2026-09-26 02:22:12.487 UTC [25255] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 02:22:12.488 UTC [25255] LOG: listening on IPv4 address "127.12.244.212", port 48499
2026-09-26 02:22:12.491 UTC [25255] LOG: listening on Unix socket "/tmp/.s.PGSQL.48499"
2026-09-26 02:22:12.517 UTC [25258] LOG: database system was shut down at 2026-09-26 02:22:10 UTC
2026-09-26 02:22:12.521 UTC [25255] LOG: database system is ready to accept connections
I20260926 02:22:13.501813 13267 mini_postgres.cc:96] Postgres bound to 48499
2026-09-26 02:22:13.519 UTC [25266] FATAL: database "slave" does not exist
127.12.244.212:48499 - accepting connections
I20260926 02:22:13.520713 13267 mini_ranger.cc:161] Starting Ranger...
I20260926 02:22:13.549147 13267 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 02:22:14.098778 13267 mini_ranger.cc:87] Created ranger Postgres database
I20260926 02:22:14.098896 13267 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-admin
2026-09-26 02:22:14,385 [I] DB FLAVOR :POSTGRES
2026-09-26 02:22:14,386 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:22:14,386 [I] Checking connection..
2026-09-26 02:22:14,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 02:22:14,634 [I] Checking connection passed.
2026-09-26 02:22:14,634 [I] --------- Verifying version history table ---------
2026-09-26 02:22:14,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:22:14,891 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:22:14,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:22:15,153 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 02:22:15,153 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 02:22:15,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 02:22:15.413 UTC [25330] WARNING: there is no transaction in progress
2026-09-26 02:22:15,420 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 02:22:15,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 02:22:15,697 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 02:22:15,697 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:22:15,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 02:22:15,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 02:22:16,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qfmw.c.gcp-upstream.internal','N') ;"
2026-09-26 02:22:16,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:22:16,743 [I] Table x_portal_user does not exist in database ranger
2026-09-26 02:22:16,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:22:16,996 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 02:22:16,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:22:17,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:22:17,522 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 02:22:17,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 02:22:18.591 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.609 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.628 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.645 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.662 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.684 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.758 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.765 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.782 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.793 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.805 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.821 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.832 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.839 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.848 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:18.855 UTC [25439] WARNING: there is no transaction in progress
2026-09-26 02:22:19,601 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 02:22:19,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 02:22:19,876 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 02:22:19,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 02:22:20,133 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 02:22:20,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 02:22:20,380 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 02:22:20,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 02:22:20,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 02:22:20,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qfmw.c.gcp-upstream.internal';"
2026-09-26 02:22:21,150 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 02:22:21,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 02:22:21,404 [I] DB_PATCHES have already been applied
I20260926 02:22:21.410959 13267 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-admin:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3VVuru/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 02:22:21.411056 13267 mini_ranger.cc:193] Using host: 127.12.244.212
I20260926 02:22:21.426298 13267 mini_ranger.cc:227] Ranger admin URL: http://127.12.244.212:48609
Sep 26, 2026 2:22:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:22:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:22:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:22:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:22:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:22:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:22:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 02:22:22.583439 13267 mini_ranger.cc:160] Time spent starting Ranger: real 9.063s user 0.013s sys 0.030s
Sep 26, 2026 2:22:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:22:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:22:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:22:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:22:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 02:22:46.061991 13267 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.244.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.244.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/rangerkms_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.244.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/rangerkms_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.244.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.244.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.244.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/HTTP_127.12.244.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 02:22:46 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389366, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 02:22:46.254833 13267 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 02:22:46.277591 13267 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 02:22:47.325001 13267 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 02:22:47.325105 13267 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms
I20260926 02:22:47.325137 13267 mini_ranger_kms.cc:227] Using postgres at 127.12.244.212:48499
2026-09-26 02:22:47,700 [I] DB FLAVOR :POSTGRES
2026-09-26 02:22:47,700 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 02:22:47,700 [I] Checking connection
2026-09-26 02:22:47,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 02:22:47,970 [I] connection success
2026-09-26 02:22:47,970 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 02:22:47,971 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 02:22:47,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 02:22:48,258 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 02:22:48,258 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 02:22:48,259 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 02:22:48,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.244.212:48499/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 02:22:48,568 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 02:22:49.125928 13267 mini_ranger_kms.cc:310] Created kms service
I20260926 02:22:49.340948 13267 mini_ranger_kms.cc:326] Created kudu user
I20260926 02:22:49.388877 13267 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 02:22:49 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389366, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:49.843401 13267 mini_ranger_kms.cc:384] Added ranger policy
I20260926 02:22:49.843735 13267 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms:/tmp/dist-test-task3VVuru/build/asan/bin/postgresql.jar:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3VVuru/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 02:22:49.843781 13267 mini_ranger_kms.cc:240] Using host: 127.12.244.212
I20260926 02:22:49.863701 13267 mini_ranger_kms.cc:276] Ranger KMS PID: 25621
I20260926 02:22:49.863870 13267 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.244.212:42067
Sep 26, 2026 2:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:22:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:22:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:22:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:22:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 02:22:52.231158 13267 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.976s user 0.086s sys 0.303s
I20260926 02:22:52.231418 13267 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 02:22:52.231602 13267 mini_ranger_kms.cc:401] 127.12.244.212:42067/kms/v1/keys
Sep 26 02:22:52 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790389372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.244.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 02:22:52 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.244.212@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
Sep 26 02:22:53 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389372, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.244.212@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
Sep 26 02:22:55 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:22:55.529368 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:37647
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:37647
--ranger_config_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:22:55.881395 25671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:55.881732 25671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:55.881881 25671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:55.889634 25671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:22:55.889765 25671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:55.889855 25671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:55.889971 25671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:22:55.890050 25671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:22:55.901026 25671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.244.212:42067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:37647
--ranger_config_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:55.902426 25671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:55.903874 25671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:55.916925 25676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:55.916882 25679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:55.916841 25677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:55.917785 25671 server_base.cc:1034] running on GCE node
I20260926 02:22:55.918675 25671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:55.920359 25671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:55.921645 25671 hybrid_clock.cc:648] HybridClock initialized: now 1790389375921562 us; error 73 us; skew 500 ppm
I20260926 02:22:55.922047 25671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:55 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:55.930555 25671 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:22:55.932359 25671 webserver.cc:460] Webserver started at http://127.12.244.254:42151/ using document root <none> and password file <none>
I20260926 02:22:55.933120 25671 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:55.933271 25671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:55.933615 25671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:55 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389375, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:55.984965 25671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "133376d319db48bda19a7b2385bb843d"
format_stamp: "Formatted at 2026-09-26 02:22:55 on dist-test-slave-qfmw"
server_key: "beab3587b10ce48b46a891659bc7b979"
server_key_iv: "4ed391986634cc71033b310e190840db"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:55.985898 25671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "133376d319db48bda19a7b2385bb843d"
format_stamp: "Formatted at 2026-09-26 02:22:55 on dist-test-slave-qfmw"
server_key: "beab3587b10ce48b46a891659bc7b979"
server_key_iv: "4ed391986634cc71033b310e190840db"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:55.991902 25671 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 02:22:56.008843 25686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:56.010578 25671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:22:56.010821 25671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "133376d319db48bda19a7b2385bb843d"
format_stamp: "Formatted at 2026-09-26 02:22:55 on dist-test-slave-qfmw"
server_key: "beab3587b10ce48b46a891659bc7b979"
server_key_iv: "4ed391986634cc71033b310e190840db"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:56.011050 25671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:56.024399 25671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:56.029531 25671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:56.029915 25671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:56.050557 25671 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:37647
I20260926 02:22:56.050559 25738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:37647 every 8 connection(s)
I20260926 02:22:56.052196 25671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:22:56.057350 25739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:22:56.057993 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 25671
I20260926 02:22:56.058212 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:22:56.059046 13267 external_mini_cluster.cc:1428] Setting key 94811fad9b26cea16c82bb4fb1ed9353
I20260926 02:22:56.077353 25739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Bootstrap starting.
I20260926 02:22:56.081321 25739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Neither blocks nor log segments found. Creating new log.
I20260926 02:22:56.082572 25739 log.cc:824] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:22:56.086463 25739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: No bootstrap required, opened a new log
Sep 26 02:22:56 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:22:56.097605 25739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:22:56.098101 25739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:22:56.098263 25739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133376d319db48bda19a7b2385bb843d, State: Initialized, Role: FOLLOWER
I20260926 02:22:56.098623 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:56.061319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.12.244.254:37647)
0926 02:22:56.071776 (+ 10457us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:56.071795 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:56.075535 (+ 3740us) server_negotiation.cc:407] Connection header received
0926 02:22:56.078466 (+ 2931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:56.078500 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:56.079186 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:56.079707 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:56.081476 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:56.083562 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:56.085119 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:56.085902 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:56.086123 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:56.090300 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:56.090309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:56.090328 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:56.090369 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:56.093915 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:56.094713 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:56.094721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:56.094730 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:56.094904 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:56.095367 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:56.095372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:56.095376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:56.095999 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:56.096323 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:56.096810 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 02:22:56.097285 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10157,"thread_start_us":9898,"threads_started":1}
I20260926 02:22:56.098977 25739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:22:56.099294 25739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:22:56.099448 25739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:22:56.099629 25739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:22:56.102983 25739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:22:56.103657 25739 leader_election.cc:302] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133376d319db48bda19a7b2385bb843d; no voters:
I20260926 02:22:56.104359 25739 leader_election.cc:288] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:22:56.104502 25744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:22:56.105909 25744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 LEADER]: Becoming Leader. State: Replica: 133376d319db48bda19a7b2385bb843d, State: Running, Role: LEADER
I20260926 02:22:56.106624 25739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:22:56.106556 25744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:22:56.110213 25746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133376d319db48bda19a7b2385bb843d. Latest consensus state: current_term: 1 leader_uuid: "133376d319db48bda19a7b2385bb843d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } } }
I20260926 02:22:56.110674 25746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: This master's current role is: LEADER
I20260926 02:22:56.110906 25745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "133376d319db48bda19a7b2385bb843d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } } }
I20260926 02:22:56.111274 25745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: This master's current role is: LEADER
I20260926 02:22:56.112066 25749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:22:56.123358 25749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:22:56.146413 25749 catalog_manager.cc:1347] Generated new cluster ID: d01b4188d45e4a1596c5d7d7ef3938f6
I20260926 02:22:56.146548 25749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:22:56.159309 25749 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:22:56.161381 25749 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:22:56.169705 25739 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 02:22:56.171586 25749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Generated new TSK 0
I20260926 02:22:56.172538 25749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:22:57.006359 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37647
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:22:57 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790389377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 02:22:57.502315 25784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:57.502854 25784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:57.503139 25784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:57.515671 25784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:57.515935 25784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:57.516187 25784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:22:57.532994 25784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.244.212:42067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:57.535007 25784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:57.536480 25784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:57.553329 25795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:57.553311 25797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:57.554154 25794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:57.554263 25784 server_base.cc:1034] running on GCE node
I20260926 02:22:57.555085 25784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:57.556432 25784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:57.557689 25784 hybrid_clock.cc:648] HybridClock initialized: now 1790389377557631 us; error 51 us; skew 500 ppm
I20260926 02:22:57.558068 25784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:57 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:57.562783 25784 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:22:57.586727 25784 webserver.cc:460] Webserver started at http://127.12.244.193:43137/ using document root <none> and password file <none>
I20260926 02:22:57.587961 25784 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:57.588152 25784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:57.588551 25784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:57 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:57.612231 25784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "1f3cd0e2966743fda29f5f618fbec042"
format_stamp: "Formatted at 2026-09-26 02:22:57 on dist-test-slave-qfmw"
server_key: "2f112aa2db0a90fa21817d27e59a714c"
server_key_iv: "d779363e8d700d288db15d30c391c605"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:57.614475 25784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "1f3cd0e2966743fda29f5f618fbec042"
format_stamp: "Formatted at 2026-09-26 02:22:57 on dist-test-slave-qfmw"
server_key: "2f112aa2db0a90fa21817d27e59a714c"
server_key_iv: "d779363e8d700d288db15d30c391c605"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:57.635906 25784 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.006s sys 0.011s
I20260926 02:22:57.685422 25804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:57.689371 25784 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.005s sys 0.016s
I20260926 02:22:57.689728 25784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "1f3cd0e2966743fda29f5f618fbec042"
format_stamp: "Formatted at 2026-09-26 02:22:57 on dist-test-slave-qfmw"
server_key: "2f112aa2db0a90fa21817d27e59a714c"
server_key_iv: "d779363e8d700d288db15d30c391c605"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:57.690136 25784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:57.712901 25784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:57.718889 25784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:57.719339 25784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:57.739578 25784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 26 02:22:57 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:57.783910 25784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:22:57.784080 25784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 02:22:57.784260 25784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:22:57.784358 25784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:57.896431 25784 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45745
I20260926 02:22:57.896471 25918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45745 every 8 connection(s)
I20260926 02:22:57.898253 25784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:22:57.900848 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 25784
I20260926 02:22:57.901062 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:22:57.901862 13267 external_mini_cluster.cc:1428] Setting key 053b0088f120bad00bab570dcfb05b66
Sep 26 02:22:57 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:22:57.943181 25921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:57.905664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37647 (local address 127.12.244.193:53647)
0926 02:22:57.917004 (+ 11340us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:57.917086 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:57.919392 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:57.921074 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:57.921087 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:57.921713 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:22:57.922572 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:57.922610 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:57.924209 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:57.924215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:57.926180 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:57.926205 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:57.927610 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:57.927617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:57.927937 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:57.928817 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:22:57.928849 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:22:57.931978 (+ 3129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:22:57.937142 (+ 5164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:57.937154 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:57.937176 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:57.938356 (+ 1180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:57.938918 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:57.938923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:57.938927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:57.939301 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:57.940131 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:22:57.940140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:22:57.940684 (+ 544us) client_negotiation.cc:769] Sending connection context
0926 02:22:57.941015 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 02:22:57.941634 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10944,"thread_start_us":14094,"threads_started":1}
I20260926 02:22:57.943828 25922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:57.906763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:53647 (local address 127.12.244.254:37647)
0926 02:22:57.919544 (+ 12781us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:57.919550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:57.919585 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:22:57.919826 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:57.919832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:57.919932 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:57.920036 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:57.922831 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:57.924010 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:57.926382 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:57.927056 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:57.927226 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:57.932386 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:57.932396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:57.932402 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:57.932455 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:57.936876 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:57.938549 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:57.938556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:57.938559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:57.938758 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:57.939465 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:57.939470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:57.939474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:57.939899 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:57.940042 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:57.943124 (+ 3082us) server_negotiation.cc:299] Negotiation successful
0926 02:22:57.943381 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12637,"thread_start_us":12481,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:22:57.945999 25919 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37647
I20260926 02:22:57.946345 25919 heartbeater.cc:464] Registering TS with master...
I20260926 02:22:57.947310 25919 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
I20260926 02:22:57.950320 25704 ts_manager.cc:194] Registered new tserver with Master: 1f3cd0e2966743fda29f5f618fbec042 (127.12.244.193:45745)
I20260926 02:22:57.952648 25704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:53647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:22:57.990372 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37647
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:22:58 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389377, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
W20260926 02:22:58.479512 25925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:58.480059 25925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:58.480263 25925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:58.488230 25925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:58.488412 25925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:58.488588 25925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:22:58.499574 25925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.244.212:42067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:58.501031 25925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:58.502468 25925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:58.517756 25939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:58.518507 25940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:58.518841 25925 server_base.cc:1034] running on GCE node
W20260926 02:22:58.519299 25942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:58.520124 25925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:58.521432 25925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:58.522686 25925 hybrid_clock.cc:648] HybridClock initialized: now 1790389378522623 us; error 54 us; skew 500 ppm
I20260926 02:22:58.523120 25925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:58 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:58.527288 25925 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:22:58.529170 25925 webserver.cc:460] Webserver started at http://127.12.244.194:39931/ using document root <none> and password file <none>
I20260926 02:22:58.530171 25925 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:58.530319 25925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:58.530694 25925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:58 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:58.549485 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "059b845728d1418898d06d1b6c138978"
format_stamp: "Formatted at 2026-09-26 02:22:58 on dist-test-slave-qfmw"
server_key: "7d12860678c957e7045d2e7012297268"
server_key_iv: "d4533983e7a0a7e3c4bee2bac14262db"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:58.550647 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "059b845728d1418898d06d1b6c138978"
format_stamp: "Formatted at 2026-09-26 02:22:58 on dist-test-slave-qfmw"
server_key: "7d12860678c957e7045d2e7012297268"
server_key_iv: "d4533983e7a0a7e3c4bee2bac14262db"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:58.558171 25925 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 02:22:58.575487 25949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:58.577272 25925 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:22:58.577497 25925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "059b845728d1418898d06d1b6c138978"
format_stamp: "Formatted at 2026-09-26 02:22:58 on dist-test-slave-qfmw"
server_key: "7d12860678c957e7045d2e7012297268"
server_key_iv: "d4533983e7a0a7e3c4bee2bac14262db"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:58.577749 25925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:58.597808 25925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:58.602483 25925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:58.602828 25925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:58.603991 25925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:22:58.605590 25925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:22:58.605703 25925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:58.605861 25925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:22:58.605933 25925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:58.646082 25925 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:43995
I20260926 02:22:58.646188 26061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:43995 every 8 connection(s)
I20260926 02:22:58.647616 25925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:22:58.653650 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 25925
I20260926 02:22:58.653851 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:22:58.654345 13267 external_mini_cluster.cc:1428] Setting key 5738ac2c52e37dcd2e77045a38035842
Sep 26 02:22:58 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389378, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:22:58.672859 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:58.650440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:52461 (local address 127.12.244.254:37647)
0926 02:22:58.650950 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:58.650957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:58.652313 (+ 1356us) server_negotiation.cc:407] Connection header received
0926 02:22:58.655475 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:58.655483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:58.655620 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:58.655755 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:58.657671 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:58.658485 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:58.660276 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:58.660721 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:58.660866 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:58.664484 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:58.664489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:58.664493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:58.664533 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:58.668259 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:58.669523 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:58.669531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:58.669534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:58.669734 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:58.670387 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:58.670393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:58.670396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:58.670827 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:58.671012 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:58.672157 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0926 02:22:58.672450 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260926 02:22:58.674275 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:58.651137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37647 (local address 127.12.244.194:52461)
0926 02:22:58.652080 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:58.652157 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:58.654880 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:58.656104 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:58.656122 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:58.656792 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:22:58.657480 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:58.657500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:58.658671 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:58.658676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:58.660101 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:58.660126 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:58.660969 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:58.660976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:58.661245 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:58.661997 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:22:58.662022 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:22:58.664207 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:22:58.668475 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:58.668499 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:58.668525 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:58.669314 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:58.669888 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:58.669896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:58.669903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:58.670210 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:58.670993 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:22:58.671006 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:22:58.671597 (+ 591us) client_negotiation.cc:769] Sending connection context
0926 02:22:58.671972 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 02:22:58.672626 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":353,"threads_started":1}
I20260926 02:22:58.676059 26062 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37647
I20260926 02:22:58.676456 26062 heartbeater.cc:464] Registering TS with master...
I20260926 02:22:58.677409 26062 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
I20260926 02:22:58.679489 25704 ts_manager.cc:194] Registered new tserver with Master: 059b845728d1418898d06d1b6c138978 (127.12.244.194:43995)
I20260926 02:22:58.680701 25704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:52461
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:22:58.718567 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37647
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:22:58.957139 25919 heartbeater.cc:503] Master 127.12.244.254:37647 was elected leader, sending a full tablet report...
W20260926 02:22:59.077674 26068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:22:59.077994 26068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:22:59.078150 26068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:22:59.086308 26068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:22:59.086450 26068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:22:59.086640 26068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:22:59.098958 26068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.244.212:42067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:22:59.100476 26068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:22:59.101948 26068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:22:59.118036 26076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:59.118000 26073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:22:59.118002 26074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:22:59.118891 26068 server_base.cc:1034] running on GCE node
I20260926 02:22:59.119628 26068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:22:59.120709 26068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:22:59.121995 26068 hybrid_clock.cc:648] HybridClock initialized: now 1790389379121881 us; error 92 us; skew 500 ppm
I20260926 02:22:59.122507 26068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:22:59 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389379, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:22:59.126500 26068 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:22:59.128326 26068 webserver.cc:460] Webserver started at http://127.12.244.195:42213/ using document root <none> and password file <none>
I20260926 02:22:59.129128 26068 fs_manager.cc:359] Metadata directory not provided
I20260926 02:22:59.129248 26068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:22:59.129606 26068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 02:22:59 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389379, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:22:59.148135 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "debf9aa79be947e99220db843770c103"
format_stamp: "Formatted at 2026-09-26 02:22:59 on dist-test-slave-qfmw"
server_key: "53e48ab0429585dec15a5a056588f0e5"
server_key_iv: "5dabda35d426ad4d0b8ce0d18c84a62d"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:59.149240 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "debf9aa79be947e99220db843770c103"
format_stamp: "Formatted at 2026-09-26 02:22:59 on dist-test-slave-qfmw"
server_key: "53e48ab0429585dec15a5a056588f0e5"
server_key_iv: "5dabda35d426ad4d0b8ce0d18c84a62d"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:59.155881 26068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:22:59.171213 26083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:59.173058 26068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:22:59.173312 26068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "debf9aa79be947e99220db843770c103"
format_stamp: "Formatted at 2026-09-26 02:22:59 on dist-test-slave-qfmw"
server_key: "53e48ab0429585dec15a5a056588f0e5"
server_key_iv: "5dabda35d426ad4d0b8ce0d18c84a62d"
server_key_version: "kuduclusterkey@0"
I20260926 02:22:59.173658 26068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:22:59.205102 26068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:22:59.209605 26068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:22:59.209940 26068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:22:59.210940 26068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:22:59.212463 26068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:22:59.212543 26068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:59.212687 26068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:22:59.212741 26068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:22:59.250787 26068 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:43655
I20260926 02:22:59.250823 26195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:43655 every 8 connection(s)
I20260926 02:22:59.252235 26068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:22:59.255652 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26068
I20260926 02:22:59.255841 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:22:59.256440 13267 external_mini_cluster.cc:1428] Setting key 79cea09a68bfaff4eb70702f4fa2dacf
Sep 26 02:22:59 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389379, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:22:59.275084 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:59.255236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:36499 (local address 127.12.244.254:37647)
0926 02:22:59.255796 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:59.255804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:59.259085 (+ 3281us) server_negotiation.cc:407] Connection header received
0926 02:22:59.259211 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:59.259215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:59.259302 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:59.259420 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:59.261069 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.261805 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.263557 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.263998 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.264167 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:59.267998 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:59.268004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:59.268008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:59.268048 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:59.271520 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.272573 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.272579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.272584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.272737 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.273242 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.273254 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.273257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.273543 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:59.273715 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:59.274460 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 02:22:59.274714 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":255,"threads_started":1}
I20260926 02:22:59.276152 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:59.255943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37647 (local address 127.12.244.195:36499)
0926 02:22:59.256865 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 02:22:59.256935 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:22:59.258779 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:22:59.259703 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:22:59.259713 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:22:59.260206 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:22:59.260848 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:59.260866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.261975 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.261979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:59.263377 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:22:59.263398 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.264252 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.264261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:22:59.264534 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:59.265347 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:22:59.265377 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:22:59.267705 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:22:59.271709 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:59.271719 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:59.271736 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:59.272438 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:59.272897 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:22:59.272901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:22:59.272903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:22:59.273105 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:22:59.273696 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:22:59.273705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:22:59.274087 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 02:22:59.274369 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 02:22:59.274933 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":309,"threads_started":1}
I20260926 02:22:59.277669 26196 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37647
I20260926 02:22:59.277966 26196 heartbeater.cc:464] Registering TS with master...
I20260926 02:22:59.278677 26196 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
I20260926 02:22:59.280258 25704 ts_manager.cc:194] Registered new tserver with Master: debf9aa79be947e99220db843770c103 (127.12.244.195:43655)
I20260926 02:22:59.281113 25704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:36499
I20260926 02:22:59.285202 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:22:59.287480 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 25784
Sep 26 02:22:59 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:22:59.305701 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:59.288416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.12.244.193:45745)
0926 02:22:59.289090 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:59.289100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:59.289135 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:22:59.289242 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:59.289251 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:59.289490 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:59.289669 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:59.291398 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.292502 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.294078 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.294729 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.294886 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:59.299577 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:59.299592 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:59.299608 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:59.299657 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:59.302555 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.303469 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.303478 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.303481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.303637 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.304043 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.304047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.304049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.304398 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:59.304737 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:59.305042 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 02:22:59.305324 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":335,"threads_started":1}
I20260926 02:22:59.307073 25914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:59.392345 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 25784
I20260926 02:22:59.409145 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 25925
Sep 26 02:22:59 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:22:59.430910 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:59.410117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.12.244.194:43995)
0926 02:22:59.410708 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:59.410715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:59.410743 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:22:59.410804 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:59.410810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:59.411048 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:59.411251 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:59.412551 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.413633 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.415255 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.415903 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.416017 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:59.425025 (+ 9008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:59.425044 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:59.425062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:59.425131 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:59.428032 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.428959 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.428968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.428971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.429141 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.429519 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.429522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.429524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.429890 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:59.430134 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:59.430384 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 02:22:59.430594 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":311,"threads_started":1}
I20260926 02:22:59.432236 26057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:59.516990 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 25925
I20260926 02:22:59.533315 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26068
Sep 26 02:22:59 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:22:59.552172 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:22:59.534439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.12.244.195:43655)
0926 02:22:59.535237 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0926 02:22:59.535244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:22:59.535275 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:22:59.535334 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:22:59.535340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:22:59.535578 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:22:59.535715 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:22:59.537126 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.538208 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.539828 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:22:59.540427 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:22:59.540539 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:22:59.545408 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:22:59.545423 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:22:59.545441 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:22:59.545489 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:22:59.548535 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.549457 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.549479 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.549482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.549634 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:22:59.550125 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:22:59.550130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:22:59.550132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:22:59.550531 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:22:59.550835 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 02:22:59.551455 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 02:22:59.551776 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":315,"threads_started":1}
I20260926 02:22:59.553589 26191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:59.639048 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26068
I20260926 02:22:59.655392 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 25671
I20260926 02:22:59.656827 25734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:22:59.767868 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 25671
I20260926 02:22:59.792323 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:37647
--webserver_interface=127.12.244.254
--webserver_port=42151
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:37647
--ranger_config_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:00.174739 26208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:00.175063 26208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:00.175246 26208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:00.182760 26208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:00.182901 26208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:00.182948 26208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:00.183010 26208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:00.183064 26208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:00.194435 26208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:37647
--ranger_config_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=42151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:00.195815 26208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:00.197293 26208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:00.210101 26213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:00.210037 26214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:00.210057 26216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:00.211239 26208 server_base.cc:1034] running on GCE node
I20260926 02:23:00.212105 26208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:00.213785 26208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:00.215034 26208 hybrid_clock.cc:648] HybridClock initialized: now 1790389380214971 us; error 61 us; skew 500 ppm
I20260926 02:23:00.215520 26208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:00 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389380, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:00.219178 26208 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:23:00.220844 26208 webserver.cc:460] Webserver started at http://127.12.244.254:42151/ using document root <none> and password file <none>
I20260926 02:23:00.221597 26208 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:00.221756 26208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:23:00 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389380, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:23:00.298399 26208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:23:00.301728 26224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:00.303494 26208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:23:00.303740 26208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "133376d319db48bda19a7b2385bb843d"
format_stamp: "Formatted at 2026-09-26 02:22:55 on dist-test-slave-qfmw"
server_key: "beab3587b10ce48b46a891659bc7b979"
server_key_iv: "4ed391986634cc71033b310e190840db"
server_key_version: "kuduclusterkey@0"
I20260926 02:23:00.304704 26208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:00.319417 26208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:00.323915 26208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:00.324185 26208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:00.343724 26208 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:37647
I20260926 02:23:00.343761 26275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:37647 every 8 connection(s)
I20260926 02:23:00.345329 26208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:23:00.348362 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26208
I20260926 02:23:00.349433 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:45745
--local_ip_for_outbound_sockets=127.12.244.193
--tserver_master_addrs=127.12.244.254:37647
--webserver_port=43137
--webserver_interface=127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:23:00.355669 26276 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:23:00.360572 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Bootstrap starting.
I20260926 02:23:00.380839 26276 log.cc:824] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:00.390722 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:23:00.391377 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Bootstrap complete.
I20260926 02:23:00.401152 26276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:23:00.402140 26276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133376d319db48bda19a7b2385bb843d, State: Initialized, Role: FOLLOWER
I20260926 02:23:00.403059 26276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:23:00.403429 26276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:23:00.403617 26276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:23:00.403877 26276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:23:00.408727 26276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:23:00.409454 26276 leader_election.cc:302] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 133376d319db48bda19a7b2385bb843d; no voters:
I20260926 02:23:00.410216 26276 leader_election.cc:288] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:23:00.410374 26280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:23:00.411595 26280 raft_consensus.cc:686] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [term 2 LEADER]: Becoming Leader. State: Replica: 133376d319db48bda19a7b2385bb843d, State: Running, Role: LEADER
I20260926 02:23:00.412249 26280 consensus_queue.cc:237] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } }
I20260926 02:23:00.412715 26276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:00.415023 26281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "133376d319db48bda19a7b2385bb843d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } } }
I20260926 02:23:00.415548 26281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:00.419531 26282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133376d319db48bda19a7b2385bb843d. Latest consensus state: current_term: 2 leader_uuid: "133376d319db48bda19a7b2385bb843d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "133376d319db48bda19a7b2385bb843d" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37647 } } }
I20260926 02:23:00.419948 26282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:00.424021 26287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:23:00.432257 26287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:23:00.434201 26287 catalog_manager.cc:1259] Loaded cluster ID: d01b4188d45e4a1596c5d7d7ef3938f6
I20260926 02:23:00.434360 26287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:23:00.440094 26287 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:23:00.442271 26287 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Loaded TSK: 0
I20260926 02:23:00.444399 26287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:23:00.446655 26276 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 02:23:00.842401 26278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:00.842844 26278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:00.843034 26278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:00.855439 26278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:00.855613 26278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:00.855816 26278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:23:00.873059 26278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:45745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=43137
--tserver_master_addrs=127.12.244.254:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:00.874908 26278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:00.876909 26278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:00.915084 26315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:00.918877 26314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:00.930065 26278 server_base.cc:1034] running on GCE node
W20260926 02:23:00.931025 26317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:00.932070 26278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:00.934297 26278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:00.937711 26278 hybrid_clock.cc:648] HybridClock initialized: now 1790389380937342 us; error 360 us; skew 500 ppm
I20260926 02:23:00.938227 26278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:00 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389380, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:00.947903 26278 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:23:00.975664 26278 webserver.cc:460] Webserver started at http://127.12.244.193:43137/ using document root <none> and password file <none>
I20260926 02:23:00.976827 26278 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:00.977028 26278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:23:00 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389380, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:23:01.026010 26278 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.010s sys 0.005s
I20260926 02:23:01.030212 26326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:01.032588 26278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:23:01.032897 26278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "1f3cd0e2966743fda29f5f618fbec042"
format_stamp: "Formatted at 2026-09-26 02:22:57 on dist-test-slave-qfmw"
server_key: "2f112aa2db0a90fa21817d27e59a714c"
server_key_iv: "d779363e8d700d288db15d30c391c605"
server_key_version: "kuduclusterkey@0"
I20260926 02:23:01.034478 26278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:01.072099 26278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:01.078697 26278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:01.079072 26278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:01.082376 26278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:01.088254 26278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:01.088441 26278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:01.088584 26278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:01.088663 26278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:01.131345 26278 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45745
I20260926 02:23:01.131372 26440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45745 every 8 connection(s)
I20260926 02:23:01.133551 26278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:23:01.142269 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26278
I20260926 02:23:01.143154 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:43995
--local_ip_for_outbound_sockets=127.12.244.194
--tserver_master_addrs=127.12.244.254:37647
--webserver_port=39931
--webserver_interface=127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:23:01 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389380, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:01.174549 26444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:01.137582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37647 (local address 127.12.244.193:32811)
0926 02:23:01.139951 (+ 2369us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:01.140038 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:01.143716 (+ 3678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:01.146259 (+ 2543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:01.146274 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:01.147020 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:01.147985 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:01.148018 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.150867 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.150875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:01.152816 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:01.152846 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.155117 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.155129 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:01.155514 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:01.156471 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:01.156511 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:01.160038 (+ 3527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:01.165671 (+ 5633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:01.165686 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:01.165712 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:01.166894 (+ 1182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:01.167864 (+ 970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:01.167874 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:01.167881 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:01.168252 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:01.169811 (+ 1559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:01.169826 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:01.170649 (+ 823us) client_negotiation.cc:769] Sending connection context
0926 02:23:01.171244 (+ 595us) client_negotiation.cc:240] Negotiation successful
0926 02:23:01.172179 (+ 935us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1911,"thread_start_us":1638,"threads_started":1}
I20260926 02:23:01.178951 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:01.137484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:32811 (local address 127.12.244.254:37647)
0926 02:23:01.140985 (+ 3501us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:01.140999 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:01.141047 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:23:01.144628 (+ 3581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:01.144654 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:01.145267 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:01.145777 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:01.148317 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.150560 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.153055 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.154381 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.154674 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:01.160528 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:01.160544 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:01.160574 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:01.160643 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:01.165395 (+ 4752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:01.167280 (+ 1885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:01.167296 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:01.167312 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:01.167643 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:01.168477 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:01.168487 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:01.168509 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:01.169554 (+ 1045us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:01.175688 (+ 6134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:01.176241 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 02:23:01.177020 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3237,"thread_start_us":2945,"threads_started":1}
I20260926 02:23:01.184616 26441 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37647
I20260926 02:23:01.185060 26441 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:01.186040 26441 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
W20260926 02:23:01.190068 26441 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:37647 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:23:01.190378 26441 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:01.190706 26441 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
I20260926 02:23:01.192328 26441 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:01.192668 26441 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
W20260926 02:23:01.193953 26441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:23:01.577636 26445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:01.577983 26445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:01.578171 26445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:01.588850 26445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:01.589133 26445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:01.589372 26445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:23:01.605962 26445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:43995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=39931
--tserver_master_addrs=127.12.244.254:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:01.608170 26445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:01.610185 26445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:01.650882 26462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:01.651590 26461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:01.658300 26445 server_base.cc:1034] running on GCE node
W20260926 02:23:01.659049 26465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:01.659938 26445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:01.661108 26445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:01.662397 26445 hybrid_clock.cc:648] HybridClock initialized: now 1790389381662324 us; error 69 us; skew 500 ppm
I20260926 02:23:01.662807 26445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:01 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:01.666368 26445 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:23:01.668623 26445 webserver.cc:460] Webserver started at http://127.12.244.194:39931/ using document root <none> and password file <none>
I20260926 02:23:01.669469 26445 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:01.669656 26445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:23:01 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:23:01.698596 26445 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:23:01.701917 26472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:01.703766 26445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
Sep 26 02:23:01 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790389381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:01.703984 26445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "059b845728d1418898d06d1b6c138978"
format_stamp: "Formatted at 2026-09-26 02:22:58 on dist-test-slave-qfmw"
server_key: "7d12860678c957e7045d2e7012297268"
server_key_iv: "d4533983e7a0a7e3c4bee2bac14262db"
server_key_version: "kuduclusterkey@0"
I20260926 02:23:01.705361 26445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:01.739475 26445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:01.745388 26445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:01.745834 26445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:01.747138 26445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:01.749030 26445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:01.749167 26445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:01.749313 26445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:01.749413 26445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:01.794631 26445 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:43995
I20260926 02:23:01.794670 26586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:43995 every 8 connection(s)
I20260926 02:23:01.796288 26445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:23:01.801812 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26445
I20260926 02:23:01.802570 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.244.212:42067/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:43655
--local_ip_for_outbound_sockets=127.12.244.195
--tserver_master_addrs=127.12.244.254:37647
--webserver_port=42213
--webserver_interface=127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:23:01 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:01.846243 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:01.802935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37647 (local address 127.12.244.194:59563)
0926 02:23:01.815770 (+ 12835us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:01.815840 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:01.823111 (+ 7271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:01.824951 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:01.824970 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:01.825544 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:01.826628 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:01.826668 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.828467 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.828475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:01.830699 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:01.830732 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.832235 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.832245 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:01.832602 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:01.833508 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:01.833547 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:01.836406 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:01.840842 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:01.840852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:01.840875 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:01.841581 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:01.842058 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:01.842063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:01.842066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:01.842332 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:01.843122 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:01.843132 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:01.843648 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 02:23:01.844007 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 02:23:01.844705 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12394,"thread_start_us":15993,"threads_started":1}
I20260926 02:23:01.847707 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:01.802072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:59563 (local address 127.12.244.254:37647)
0926 02:23:01.824091 (+ 22019us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:01.824097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:01.824119 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:23:01.824187 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:01.824192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:01.824283 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:01.824405 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:01.826980 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.828186 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.830955 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:01.831685 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:01.831824 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:01.836771 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:01.836779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:01.836783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:01.836830 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:01.840575 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:01.841761 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:01.841767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:01.841771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:01.841928 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:01.842475 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:01.842480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:01.842483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:01.842870 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:01.843009 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:01.846799 (+ 3790us) server_negotiation.cc:299] Negotiation successful
0926 02:23:01.847070 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21878,"thread_start_us":21732,"threads_started":1}
I20260926 02:23:01.850361 26587 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37647
I20260926 02:23:01.850775 26587 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:01.851981 26587 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
I20260926 02:23:01.854703 26241 ts_manager.cc:194] Registered new tserver with Master: 059b845728d1418898d06d1b6c138978 (127.12.244.194:43995)
I20260926 02:23:01.857276 26241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:59563
I20260926 02:23:02.194108 26441 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:02.194548 26441 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
I20260926 02:23:02.195922 26241 ts_manager.cc:194] Registered new tserver with Master: 1f3cd0e2966743fda29f5f618fbec042 (127.12.244.193:45745)
I20260926 02:23:02.197382 26241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:32811
Sep 26 02:23:02 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
W20260926 02:23:02.331799 26591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:02.332304 26591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:02.332469 26591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:02.344398 26591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:02.344594 26591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:02.344827 26591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:23:02.357707 26591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.244.212:42067/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:43655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=42213
--tserver_master_addrs=127.12.244.254:37647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:02.359246 26591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:02.360867 26591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:02.383827 26596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:02.386585 26597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:02.389694 26599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:02.390024 26591 server_base.cc:1034] running on GCE node
I20260926 02:23:02.391034 26591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:02.392607 26591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:02.394085 26591 hybrid_clock.cc:648] HybridClock initialized: now 1790389382393934 us; error 146 us; skew 500 ppm
I20260926 02:23:02.394606 26591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:02 dist-test-slave-qfmw krb5kdc[25224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:02.398710 26591 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:23:02.401029 26591 webserver.cc:460] Webserver started at http://127.12.244.195:42213/ using document root <none> and password file <none>
I20260926 02:23:02.401914 26591 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:02.402108 26591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 02:23:02 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:23:02.437278 26591 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.006s sys 0.007s
I20260926 02:23:02.441242 26607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:02.443378 26591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:23:02.443686 26591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "debf9aa79be947e99220db843770c103"
format_stamp: "Formatted at 2026-09-26 02:22:59 on dist-test-slave-qfmw"
server_key: "53e48ab0429585dec15a5a056588f0e5"
server_key_iv: "5dabda35d426ad4d0b8ce0d18c84a62d"
server_key_version: "kuduclusterkey@0"
I20260926 02:23:02.444978 26591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:02.483338 26591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:02.488566 26591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:02.488955 26591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:02.490325 26591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:02.492246 26591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:02.492383 26591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:02.492544 26591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:02.492658 26591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:23:02 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389381, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for HTTP/127.12.244.212@KRBTEST.COM
I20260926 02:23:02.625645 26591 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:43655
I20260926 02:23:02.625684 26721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:43655 every 8 connection(s)
I20260926 02:23:02.627985 26591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:23:02.636873 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26591
Sep 26 02:23:02 dist-test-slave-qfmw krb5kdc[25224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389382, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:02.673408 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.638756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.12.244.254:37647)
0926 02:23:02.644872 (+ 6116us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:02.644880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:02.646658 (+ 1778us) server_negotiation.cc:407] Connection header received
0926 02:23:02.646842 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:02.646848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:02.646949 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:02.647054 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:02.655027 (+ 7973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.656361 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.658434 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.659199 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.659334 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.663790 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:02.663798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:02.663803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:02.663864 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:02.667753 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:02.668907 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:02.668916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:02.668919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:02.669112 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:02.670102 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:02.670113 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:02.670117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:02.670923 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:02.671269 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:02.672614 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0926 02:23:02.672863 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5973,"thread_start_us":5828,"threads_started":1}
I20260926 02:23:02.678182 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.632072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:38213 (local address 127.12.244.254:37647)
0926 02:23:02.632672 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:02.632680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:02.634954 (+ 2274us) server_negotiation.cc:407] Connection header received
0926 02:23:02.645939 (+ 10985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:02.645945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:02.646046 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:02.646180 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:02.648652 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.649852 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.657315 (+ 7463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.657969 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.658079 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.668532 (+ 10453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:02.668539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:02.668544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:02.668591 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:02.671969 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:02.672970 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:02.672975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:02.672979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:02.673131 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:02.673673 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:02.673678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:02.673681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:02.674004 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:02.674136 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:02.677473 (+ 3337us) server_negotiation.cc:299] Negotiation successful
0926 02:23:02.677779 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":310,"threads_started":1}
I20260926 02:23:02.681277 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.632859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37647 (local address 127.12.244.195:38213)
0926 02:23:02.634765 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:02.634836 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:02.645562 (+ 10726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:02.646727 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:02.646743 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:02.647496 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:02.648421 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.648463 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.653974 (+ 5511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.653981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.656291 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.656314 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.662231 (+ 5917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.662239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.662592 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.665209 (+ 2617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:02.665246 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:02.668195 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:02.672180 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:02.672191 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:02.672214 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:02.672833 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:02.673269 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:02.673273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:02.673276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:02.673551 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:02.674228 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:02.674237 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:02.674735 (+ 498us) client_negotiation.cc:769] Sending connection context
0926 02:23:02.678852 (+ 4117us) client_negotiation.cc:240] Negotiation successful
0926 02:23:02.679627 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"thread_start_us":1303,"threads_started":1}
I20260926 02:23:02.683732 26722 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37647
I20260926 02:23:02.684123 26722 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:02.685115 26722 heartbeater.cc:511] Master 127.12.244.254:37647 requested a full tablet report, sending...
I20260926 02:23:02.687055 26241 ts_manager.cc:194] Registered new tserver with Master: debf9aa79be947e99220db843770c103 (127.12.244.195:43655)
I20260926 02:23:02.688288 26241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:38213
I20260926 02:23:02.692562 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:23:02.727862 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.696861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.12.244.254:37647)
0926 02:23:02.697041 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:02.697046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:02.708727 (+ 11681us) server_negotiation.cc:407] Connection header received
0926 02:23:02.709040 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:02.709047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:02.709173 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:02.709252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:02.711692 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.712732 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.715144 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.715920 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.716097 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.720211 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:02.720219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:02.720223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:02.720278 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:02.724103 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:02.725272 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:02.725279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:02.725283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:02.725494 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:02.726008 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:02.726014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:02.726018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:02.726391 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:02.726633 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:02.727263 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 02:23:02.727522 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:23:02.735858 26241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:23:02.748420 26241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:23:02.810998 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.780389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.12.244.193:45745)
0926 02:23:02.793726 (+ 13337us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:02.793737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:02.798806 (+ 5069us) server_negotiation.cc:407] Connection header received
0926 02:23:02.798910 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:02.798919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:02.799252 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:02.799453 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:02.800715 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.802542 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.805961 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.807966 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.808252 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.808507 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:02.810378 (+ 1871us) server_negotiation.cc:299] Negotiation successful
0926 02:23:02.810579 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13135,"thread_start_us":12953,"threads_started":1}
I20260926 02:23:02.811384 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.780261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45745 (local address 127.0.0.1:56248)
0926 02:23:02.798373 (+ 18112us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:02.798417 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:02.798607 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:02.799789 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:02.799801 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:02.799855 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:02.800417 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.800434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.802776 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.802785 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.805669 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.805696 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.808690 (+ 2994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.808697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.808924 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.808996 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 02:23:02.809152 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 02:23:02.809453 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17791,"thread_start_us":17604,"threads_started":1}
I20260926 02:23:02.820530 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.798913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:43655 (local address 127.0.0.1:35732)
0926 02:23:02.811845 (+ 12932us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:02.811865 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:02.812004 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:02.813339 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:02.813344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:02.813373 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:02.813617 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.813629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.815868 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.815874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.817655 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.817677 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.819740 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.819747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.819923 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.819963 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:23:02.820051 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 02:23:02.820235 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12784,"thread_start_us":19018,"threads_started":1}
I20260926 02:23:02.821770 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.799301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.12.244.195:43655)
0926 02:23:02.812638 (+ 13337us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:02.812647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:02.812702 (+ 55us) server_negotiation.cc:407] Connection header received
0926 02:23:02.812806 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:02.812814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:02.813107 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:02.813806 (+ 699us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:02.814102 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.815697 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.817932 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.819530 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.820953 (+ 1423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.821174 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:02.821250 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 02:23:02.821410 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13152,"thread_start_us":19801,"threads_started":1}
I20260926 02:23:02.827620 26656 tablet_service.cc:1467] Processing CreateTablet for tablet b74014051fd2421c9d97e0362fcd1458 (DEFAULT_TABLE table=test-table [id=40c40db0eec146e5bb4121db3219ffce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:02.829566 26656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74014051fd2421c9d97e0362fcd1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:02.831194 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.822520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:43995 (local address 127.0.0.1:59398)
0926 02:23:02.822665 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:02.822686 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:02.822801 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:02.823813 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:02.823818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:02.823844 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:02.824045 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.824058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.826344 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.826350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.828299 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:02.828320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.830307 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.830315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:02.830535 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.830573 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:23:02.830655 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 02:23:02.830818 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 02:23:02.834260 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:02.818892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.12.244.194:43995)
0926 02:23:02.822479 (+ 3587us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:02.822488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:02.823123 (+ 635us) server_negotiation.cc:407] Connection header received
0926 02:23:02.823255 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:02.823265 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:02.823579 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:02.824237 (+ 658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:02.824518 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.826144 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.828577 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:02.830092 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:02.831847 (+ 1755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:02.832040 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:02.832134 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 02:23:02.832341 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3411,"thread_start_us":3266,"threads_started":1}
I20260926 02:23:02.838971 26520 tablet_service.cc:1467] Processing CreateTablet for tablet b74014051fd2421c9d97e0362fcd1458 (DEFAULT_TABLE table=test-table [id=40c40db0eec146e5bb4121db3219ffce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:02.838972 26375 tablet_service.cc:1467] Processing CreateTablet for tablet b74014051fd2421c9d97e0362fcd1458 (DEFAULT_TABLE table=test-table [id=40c40db0eec146e5bb4121db3219ffce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:02.840677 26375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74014051fd2421c9d97e0362fcd1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:02.840873 26520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b74014051fd2421c9d97e0362fcd1458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:02.863015 26587 heartbeater.cc:503] Master 127.12.244.254:37647 was elected leader, sending a full tablet report...
I20260926 02:23:02.866185 26742 tablet_bootstrap.cc:492] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Bootstrap starting.
I20260926 02:23:02.871894 26743 tablet_bootstrap.cc:492] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Bootstrap starting.
I20260926 02:23:02.869869 26744 tablet_bootstrap.cc:492] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Bootstrap starting.
I20260926 02:23:02.874917 26742 tablet_bootstrap.cc:654] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:02.875912 26743 tablet_bootstrap.cc:654] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:02.876524 26742 log.cc:824] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:02.876585 26744 tablet_bootstrap.cc:654] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:02.878082 26744 log.cc:824] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:02.885731 26743 log.cc:824] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:02.887729 26742 tablet_bootstrap.cc:492] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: No bootstrap required, opened a new log
I20260926 02:23:02.888259 26742 ts_tablet_manager.cc:1397] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.004s
I20260926 02:23:02.896775 26744 tablet_bootstrap.cc:492] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: No bootstrap required, opened a new log
I20260926 02:23:02.897243 26744 ts_tablet_manager.cc:1397] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.014s
I20260926 02:23:02.898043 26743 tablet_bootstrap.cc:492] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: No bootstrap required, opened a new log
I20260926 02:23:02.898350 26743 ts_tablet_manager.cc:1397] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20260926 02:23:02.908495 26744 raft_consensus.cc:348] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:02.909201 26744 raft_consensus.cc:374] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:02.909358 26744 raft_consensus.cc:729] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: debf9aa79be947e99220db843770c103, State: Initialized, Role: FOLLOWER
I20260926 02:23:02.910228 26744 consensus_queue.cc:260] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:02.910550 26742 raft_consensus.cc:348] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:02.911283 26742 raft_consensus.cc:374] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:02.911051 26743 raft_consensus.cc:348] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:02.911427 26742 raft_consensus.cc:729] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059b845728d1418898d06d1b6c138978, State: Initialized, Role: FOLLOWER
I20260926 02:23:02.911594 26743 raft_consensus.cc:374] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:02.911717 26743 raft_consensus.cc:729] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3cd0e2966743fda29f5f618fbec042, State: Initialized, Role: FOLLOWER
I20260926 02:23:02.912400 26743 consensus_queue.cc:260] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:02.912469 26742 consensus_queue.cc:260] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:02.922344 26743 ts_tablet_manager.cc:1428] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Time spent starting tablet: real 0.024s user 0.019s sys 0.005s
I20260926 02:23:02.923650 26441 heartbeater.cc:503] Master 127.12.244.254:37647 was elected leader, sending a full tablet report...
I20260926 02:23:02.925369 26722 heartbeater.cc:503] Master 127.12.244.254:37647 was elected leader, sending a full tablet report...
I20260926 02:23:02.925434 26742 ts_tablet_manager.cc:1428] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Time spent starting tablet: real 0.037s user 0.018s sys 0.004s
I20260926 02:23:02.926208 26744 ts_tablet_manager.cc:1428] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Time spent starting tablet: real 0.029s user 0.016s sys 0.007s
W20260926 02:23:03.071161 26588 tablet.cc:2369] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:03.089901 26753 raft_consensus.cc:482] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:23:03.090301 26753 raft_consensus.cc:504] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:03.092312 26753 leader_election.cc:288] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f3cd0e2966743fda29f5f618fbec042 (127.12.244.193:45745), debf9aa79be947e99220db843770c103 (127.12.244.195:43655)
I20260926 02:23:03.100776 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.092832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:57047 (local address 127.12.244.193:45745)
0926 02:23:03.093114 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:03.093124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:03.093804 (+ 680us) server_negotiation.cc:407] Connection header received
0926 02:23:03.094160 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:03.094169 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:03.094280 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:03.094362 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:03.095470 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.096500 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.098760 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.099674 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.099881 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.099967 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:03.100384 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 02:23:03.100524 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:23:03.101029 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.092692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45745 (local address 127.12.244.194:57047)
0926 02:23:03.093674 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:03.093701 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:03.093917 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:03.094628 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:03.094636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:03.094676 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:03.095262 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:03.095284 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.096645 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.096650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:03.098580 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:03.098601 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.099919 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.099925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:03.100197 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.100227 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:23:03.100354 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 02:23:03.100503 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":55,"thread_start_us":605,"threads_started":1}
I20260926 02:23:03.102288 26395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74014051fd2421c9d97e0362fcd1458" candidate_uuid: "059b845728d1418898d06d1b6c138978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f3cd0e2966743fda29f5f618fbec042" is_pre_election: true
I20260926 02:23:03.102851 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.094082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:54939 (local address 127.12.244.195:43655)
0926 02:23:03.094257 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:03.094263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:03.095796 (+ 1533us) server_negotiation.cc:407] Connection header received
0926 02:23:03.095961 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:03.095967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:03.096044 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:03.096141 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:03.096759 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.097836 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.100513 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.101775 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.101960 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.102038 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:03.102435 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 02:23:03.102587 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:23:03.103008 26395 raft_consensus.cc:2454] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059b845728d1418898d06d1b6c138978 in term 0.
I20260926 02:23:03.102918 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.093858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:43655 (local address 127.12.244.194:54939)
0926 02:23:03.095649 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:03.095670 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:03.095813 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:03.096186 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:03.096192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:03.096219 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:03.096573 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:03.096585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.098010 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.098016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:03.100301 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:03.100319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.102007 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.102014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:03.102202 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.102238 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:23:03.102351 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:23:03.102519 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"mutex_wait_us":30,"thread_start_us":1452,"threads_started":1}
I20260926 02:23:03.104105 26473 leader_election.cc:302] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059b845728d1418898d06d1b6c138978, 1f3cd0e2966743fda29f5f618fbec042; no voters:
I20260926 02:23:03.104611 26753 raft_consensus.cc:2781] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:23:03.104802 26753 raft_consensus.cc:482] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:23:03.104964 26753 raft_consensus.cc:3037] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:03.104820 26676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74014051fd2421c9d97e0362fcd1458" candidate_uuid: "059b845728d1418898d06d1b6c138978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debf9aa79be947e99220db843770c103" is_pre_election: true
I20260926 02:23:03.105474 26676 raft_consensus.cc:2454] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 059b845728d1418898d06d1b6c138978 in term 0.
I20260926 02:23:03.107972 26753 raft_consensus.cc:504] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:03.108779 26753 leader_election.cc:288] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [CANDIDATE]: Term 1 election: Requested vote from peers 1f3cd0e2966743fda29f5f618fbec042 (127.12.244.193:45745), debf9aa79be947e99220db843770c103 (127.12.244.195:43655)
I20260926 02:23:03.109412 26395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74014051fd2421c9d97e0362fcd1458" candidate_uuid: "059b845728d1418898d06d1b6c138978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f3cd0e2966743fda29f5f618fbec042"
I20260926 02:23:03.109508 26676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b74014051fd2421c9d97e0362fcd1458" candidate_uuid: "059b845728d1418898d06d1b6c138978" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "debf9aa79be947e99220db843770c103"
I20260926 02:23:03.109728 26395 raft_consensus.cc:3037] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:03.109763 26676 raft_consensus.cc:3037] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:03.112996 26395 raft_consensus.cc:2454] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059b845728d1418898d06d1b6c138978 in term 1.
I20260926 02:23:03.113476 26676 raft_consensus.cc:2454] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 059b845728d1418898d06d1b6c138978 in term 1.
I20260926 02:23:03.113726 26473 leader_election.cc:302] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 059b845728d1418898d06d1b6c138978, 1f3cd0e2966743fda29f5f618fbec042; no voters:
I20260926 02:23:03.114145 26753 raft_consensus.cc:2781] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:03.114719 26753 raft_consensus.cc:686] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 1 LEADER]: Becoming Leader. State: Replica: 059b845728d1418898d06d1b6c138978, State: Running, Role: LEADER
I20260926 02:23:03.115667 26753 consensus_queue.cc:237] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } }
I20260926 02:23:03.121575 26239 catalog_manager.cc:5495] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 reported cstate change: term changed from 0 to 1, leader changed from <none> to 059b845728d1418898d06d1b6c138978 (127.12.244.194). New cstate: current_term: 1 leader_uuid: "059b845728d1418898d06d1b6c138978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059b845728d1418898d06d1b6c138978" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 43995 } health_report { overall_health: HEALTHY } } }
W20260926 02:23:03.130842 26723 tablet.cc:2369] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:23:03.136932 26442 tablet.cc:2369] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:03.165266 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.159166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.12.244.194:43995)
0926 02:23:03.159394 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:03.159401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:03.159544 (+ 143us) server_negotiation.cc:407] Connection header received
0926 02:23:03.159842 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:03.159847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:03.159948 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:03.160134 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:23:03.160611 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.161370 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.162849 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.163356 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.163499 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.164077 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:23:03.164623 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:23:03.164816 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:03.164874 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 02:23:03.164965 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:23:03.174780 26395 raft_consensus.cc:1261] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 1 FOLLOWER]: Refusing update from remote peer 059b845728d1418898d06d1b6c138978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:03.174993 26676 raft_consensus.cc:1261] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 1 FOLLOWER]: Refusing update from remote peer 059b845728d1418898d06d1b6c138978: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:03.175976 26753 consensus_queue.cc:1035] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f3cd0e2966743fda29f5f618fbec042" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:03.176317 26760 consensus_queue.cc:1035] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "debf9aa79be947e99220db843770c103" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 43655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:03.204926 26240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:03.205281 26240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:03.209436 26240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 133376d319db48bda19a7b2385bb843d: Sending DeleteTablet for 3 replicas of tablet b74014051fd2421c9d97e0362fcd1458
I20260926 02:23:03.211037 26375 tablet_service.cc:1514] Processing DeleteTablet for tablet b74014051fd2421c9d97e0362fcd1458 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:03 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:56248
I20260926 02:23:03.211158 26656 tablet_service.cc:1514] Processing DeleteTablet for tablet b74014051fd2421c9d97e0362fcd1458 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:03 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:35732
I20260926 02:23:03.211506 26520 tablet_service.cc:1514] Processing DeleteTablet for tablet b74014051fd2421c9d97e0362fcd1458 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:03 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:59398
I20260926 02:23:03.218590 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26278
I20260926 02:23:03.227624 26773 tablet_replica.cc:330] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: stopping tablet replica
I20260926 02:23:03.228641 26772 tablet_replica.cc:330] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: stopping tablet replica
I20260926 02:23:03.229205 26773 raft_consensus.cc:2229] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:03.229204 26772 raft_consensus.cc:2229] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:03.229820 26772 raft_consensus.cc:2258] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:03.230051 26774 tablet_replica.cc:330] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: stopping tablet replica
I20260926 02:23:03.230510 26774 raft_consensus.cc:2229] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:23:03.230801 26773 raft_consensus.cc:2258] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:03.231942 26774 raft_consensus.cc:2258] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:03.233714 26773 ts_tablet_manager.cc:1905] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:03.233712 26772 ts_tablet_manager.cc:1905] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:03.241813 26773 ts_tablet_manager.cc:1918] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:03.242111 26773 log.cc:1196] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/b74014051fd2421c9d97e0362fcd1458
I20260926 02:23:03.242859 26773 ts_tablet_manager.cc:1939] T b74014051fd2421c9d97e0362fcd1458 P debf9aa79be947e99220db843770c103: Deleting consensus metadata
I20260926 02:23:03.242880 26774 ts_tablet_manager.cc:1905] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:03.245222 26225 catalog_manager.cc:4841] TS debf9aa79be947e99220db843770c103 (127.12.244.195:43655): tablet b74014051fd2421c9d97e0362fcd1458 (table test-table [id=40c40db0eec146e5bb4121db3219ffce]) successfully deleted
I20260926 02:23:03.247983 26772 ts_tablet_manager.cc:1918] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:03.248277 26772 log.cc:1196] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/b74014051fd2421c9d97e0362fcd1458
I20260926 02:23:03.248991 26772 ts_tablet_manager.cc:1939] T b74014051fd2421c9d97e0362fcd1458 P 1f3cd0e2966743fda29f5f618fbec042: Deleting consensus metadata
I20260926 02:23:03.252903 26225 catalog_manager.cc:4841] TS 1f3cd0e2966743fda29f5f618fbec042 (127.12.244.193:45745): tablet b74014051fd2421c9d97e0362fcd1458 (table test-table [id=40c40db0eec146e5bb4121db3219ffce]) successfully deleted
I20260926 02:23:03.254501 26774 ts_tablet_manager.cc:1918] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:03.254578 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.220039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.12.244.193:45745)
0926 02:23:03.220270 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:03.220280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:03.227874 (+ 7594us) server_negotiation.cc:407] Connection header received
0926 02:23:03.228111 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:03.228120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:03.228250 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:03.228416 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:03.231385 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.232566 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.240678 (+ 8112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.241423 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.241648 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.245669 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:03.245688 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:03.245708 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:03.245762 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:03.250180 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:03.251285 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:03.251296 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:03.251300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:03.251500 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:03.252022 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:03.252027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:03.252031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:03.252548 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:03.253459 (+ 911us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:03.253868 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 02:23:03.254162 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:23:03.254792 26774 log.cc:1196] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/b74014051fd2421c9d97e0362fcd1458
I20260926 02:23:03.255405 26435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:03.255558 26774 ts_tablet_manager.cc:1939] T b74014051fd2421c9d97e0362fcd1458 P 059b845728d1418898d06d1b6c138978: Deleting consensus metadata
I20260926 02:23:03.258183 26225 catalog_manager.cc:4841] TS 059b845728d1418898d06d1b6c138978 (127.12.244.194:43995): tablet b74014051fd2421c9d97e0362fcd1458 (table test-table [id=40c40db0eec146e5bb4121db3219ffce]) successfully deleted
I20260926 02:23:03.349633 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26278
I20260926 02:23:03.376684 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26445
I20260926 02:23:03.395383 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.377691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.12.244.194:43995)
0926 02:23:03.377867 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:03.377871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:03.377890 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:23:03.377949 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:03.377953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:03.378036 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:03.378124 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:03.380268 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.381151 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.383644 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.384146 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.384261 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.388130 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:03.388147 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:03.388165 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:03.388218 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:03.391748 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:03.392845 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:03.392854 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:03.392856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:03.393036 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:03.393561 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:03.393566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:03.393569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:03.394012 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:03.394259 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:03.394736 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 02:23:03.394945 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:23:03.396286 26582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:03.509668 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26445
I20260926 02:23:03.528173 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26591
I20260926 02:23:03.543489 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:03.528991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.12.244.195:43655)
0926 02:23:03.529224 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:03.529229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:03.529268 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:23:03.529477 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:03.529482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:03.529559 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:03.529656 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:03.531231 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.531934 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.533368 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:03.533767 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:03.533870 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:03.536864 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:03.536882 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:03.536903 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:03.536954 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:03.540333 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:03.541287 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:03.541296 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:03.541298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:03.541454 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:03.541935 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:03.541939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:03.541941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:03.542325 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:03.542547 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:03.542862 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 02:23:03.543040 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:23:03.544307 26716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:03.634517 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26591
I20260926 02:23:03.650990 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26208
I20260926 02:23:03.652392 26271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:03.765300 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26208
I20260926 02:23:03.788152 13267 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 02:23:04.124534 13267 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 02:23:04.124755 13267 mini_ranger.cc:66] Stopping Ranger...
I20260926 02:23:04.471233 13267 mini_ranger.cc:68] Stopped Ranger
2026-09-26 02:23:04.471 UTC [25255] LOG: received smart shutdown request
2026-09-26 02:23:04.474 UTC [25255] LOG: background worker "logical replication launcher" (PID 25264) exited with exit code 1
2026-09-26 02:23:04.474 UTC [25259] LOG: shutting down
2026-09-26 02:23:04.662 UTC [25255] LOG: database system is shut down
2026-09-26T02:23:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56610 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:23:04 dist-test-slave-qfmw krb5kdc[26788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:23:04 dist-test-slave-qfmw krb5kdc[26788](info): set up 2 sockets
Sep 26 02:23:04 dist-test-slave-qfmw krb5kdc[26788](info): commencing operation
krb5kdc: starting...
W20260926 02:23:05.910220 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:23:06 dist-test-slave-qfmw krb5kdc[26788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:23:06.159911 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39979
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39979,127.12.244.253:35993,127.12.244.252:44429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:06.518652 26801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:06.519053 26801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:06.519240 26801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:06.527001 26801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:06.527216 26801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:06.527341 26801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:06.527498 26801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:06.527591 26801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:06.538337 26801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39979,127.12.244.253:35993,127.12.244.252:44429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:06.539942 26801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:06.541350 26801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:06.554549 26809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:06.554579 26807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:06.554500 26806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:06.555827 26801 server_base.cc:1034] running on GCE node
I20260926 02:23:06.556701 26801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:06.558282 26801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:06.559525 26801 hybrid_clock.cc:648] HybridClock initialized: now 1790389386559473 us; error 63 us; skew 500 ppm
I20260926 02:23:06.559916 26801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:06 dist-test-slave-qfmw krb5kdc[26788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:06.570796 26801 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:23:06.572664 26801 webserver.cc:460] Webserver started at http://127.12.244.254:37529/ using document root <none> and password file <none>
I20260926 02:23:06.573415 26801 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:06.573525 26801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:06.573792 26801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:06.577163 26801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "4407a95db22f4774bee77a2aadbd8e21"
format_stamp: "Formatted at 2026-09-26 02:23:06 on dist-test-slave-qfmw"
server_key: "40d75f6626481ed6e0795a9b4186f165"
server_key_iv: "d1e2806292c44afb43237c36af756037"
server_key_version: "encryptionkey@0"
I20260926 02:23:06.578027 26801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "4407a95db22f4774bee77a2aadbd8e21"
format_stamp: "Formatted at 2026-09-26 02:23:06 on dist-test-slave-qfmw"
server_key: "40d75f6626481ed6e0795a9b4186f165"
server_key_iv: "d1e2806292c44afb43237c36af756037"
server_key_version: "encryptionkey@0"
I20260926 02:23:06.584000 26801 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260926 02:23:06.588197 26816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:06.589956 26801 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:23:06.590167 26801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "4407a95db22f4774bee77a2aadbd8e21"
format_stamp: "Formatted at 2026-09-26 02:23:06 on dist-test-slave-qfmw"
server_key: "40d75f6626481ed6e0795a9b4186f165"
server_key_iv: "d1e2806292c44afb43237c36af756037"
server_key_version: "encryptionkey@0"
I20260926 02:23:06.590391 26801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:06.617625 26801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:06.623245 26801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:06.623636 26801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:06.643462 26801 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:39979
I20260926 02:23:06.643498 26868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:39979 every 8 connection(s)
I20260926 02:23:06.644961 26801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:23:06.649075 26869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:06.655157 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26801
I20260926 02:23:06.655400 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:23:06.656057 13267 external_mini_cluster.cc:1428] Setting key 6afd754c0c6234fcca5370b16bacdb4f
I20260926 02:23:06.656297 26869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:23:06 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.244.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.253@KRBTEST.COM" created.
I20260926 02:23:06.681948 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.659882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.12.244.254:39979)
0926 02:23:06.661864 (+ 1982us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:06.661875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:06.661935 (+ 60us) server_negotiation.cc:407] Connection header received
0926 02:23:06.663761 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:06.663768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:06.664024 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:06.664265 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:06.665687 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:06.666939 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:06.668479 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:06.669096 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:06.669338 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:06.673309 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:06.673318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:06.673334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:06.673370 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:06.676780 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:06.677608 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:06.677616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:06.677639 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:06.677895 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:06.678315 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:06.678320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:06.678323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:06.679011 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:06.679261 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:06.679889 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 02:23:06.680271 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1755,"spinlock_wait_cycles":23808,"thread_start_us":1516,"threads_started":1}
I20260926 02:23:06.681927 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.659888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39979 (local address 127.0.0.1:49498)
0926 02:23:06.661114 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.661182 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:06.663330 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:06.664283 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:06.664290 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:06.665093 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:06.665465 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:06.665485 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:06.667136 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:06.667144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:06.668317 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:06.668336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:06.669365 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:06.669371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:06.669548 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:06.670316 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:06.670346 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:06.673048 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:06.676952 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:06.676967 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:06.676989 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:06.677442 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:06.678020 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:06.678026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:06.678029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:06.678196 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:06.679179 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:06.679187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:06.679472 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 02:23:06.679751 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 02:23:06.680402 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":161,"thread_start_us":490,"threads_started":1}
I20260926 02:23:06.685803 26869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:23:06.687265 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.686717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36568)
0926 02:23:06.686928 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.687051 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260926 02:23:06.687628 26817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
I20260926 02:23:06.689494 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.689073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36572)
0926 02:23:06.689243 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.689336 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 02:23:06.689728 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:35993: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab.
Entry for principal kudu/127.12.244.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab.
I20260926 02:23:06.714802 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:35993
--webserver_interface=127.12.244.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab
--principal=kudu/127.12.244.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39979,127.12.244.253:35993,127.12.244.252:44429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:23:06.739127 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } attempt: 1
I20260926 02:23:06.740319 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.739835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36586)
0926 02:23:06.740042 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.740184 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:23:06.741359 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.741003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36592)
0926 02:23:06.741150 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.741232 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:23:06.741539 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:35993: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
I20260926 02:23:06.809827 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } attempt: 2
I20260926 02:23:06.811152 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.810626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36608)
0926 02:23:06.810873 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.810969 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:23:06.812274 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.811864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36612)
0926 02:23:06.812037 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.812120 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 02:23:06.812454 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:35993: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
I20260926 02:23:06.903720 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } attempt: 3
I20260926 02:23:06.905076 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.904437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36614)
0926 02:23:06.904710 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.904869 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 02:23:06.906422 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:06.905878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36616)
0926 02:23:06.906074 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:06.906196 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:23:06.906666 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:35993: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
I20260926 02:23:07.049961 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } attempt: 4
I20260926 02:23:07.051433 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.050700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36622)
0926 02:23:07.051076 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.051258 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260926 02:23:07.052686 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.052227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36630)
0926 02:23:07.052398 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.052514 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:23:07.053083 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:35993: Network error: Client connection negotiation failed: client connection to 127.12.244.253:35993: connect: Connection refused (error 111)
W20260926 02:23:07.064534 26876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:07.064868 26876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:07.065025 26876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:07.072836 26876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:07.072968 26876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:07.073051 26876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:07.073148 26876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:07.073231 26876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:07.083822 26876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39979,127.12.244.253:35993,127.12.244.252:44429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab
--principal=kudu/127.12.244.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:35993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.244.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:07.085299 26876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:07.086748 26876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:07.100047 26881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:07.100193 26884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:07.100171 26882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:07.100857 26876 server_base.cc:1034] running on GCE node
I20260926 02:23:07.101835 26876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:07.103546 26876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:07.104776 26876 hybrid_clock.cc:648] HybridClock initialized: now 1790389387104725 us; error 39 us; skew 500 ppm
I20260926 02:23:07.105189 26876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:07.108597 26876 init.cc:377] Logged in from keytab as kudu/127.12.244.253@KRBTEST.COM (short username kudu)
I20260926 02:23:07.110194 26876 webserver.cc:460] Webserver started at http://127.12.244.253:45731/ using document root <none> and password file <none>
I20260926 02:23:07.110953 26876 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:07.111104 26876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:07.111490 26876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:07.114656 26876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data/instance:
uuid: "a138a2a3de4c4f26892fcade34338707"
format_stamp: "Formatted at 2026-09-26 02:23:07 on dist-test-slave-qfmw"
server_key: "0647e1b5da43f567fdf3996e825523c4"
server_key_iv: "3c6cd833074c95b8005ca490818e3e8b"
server_key_version: "encryptionkey@0"
I20260926 02:23:07.115577 26876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/wal/instance:
uuid: "a138a2a3de4c4f26892fcade34338707"
format_stamp: "Formatted at 2026-09-26 02:23:07 on dist-test-slave-qfmw"
server_key: "0647e1b5da43f567fdf3996e825523c4"
server_key_iv: "3c6cd833074c95b8005ca490818e3e8b"
server_key_version: "encryptionkey@0"
I20260926 02:23:07.121258 26876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:23:07.125339 26891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:07.127364 26876 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:23:07.127596 26876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/wal
uuid: "a138a2a3de4c4f26892fcade34338707"
format_stamp: "Formatted at 2026-09-26 02:23:07 on dist-test-slave-qfmw"
server_key: "0647e1b5da43f567fdf3996e825523c4"
server_key_iv: "3c6cd833074c95b8005ca490818e3e8b"
server_key_version: "encryptionkey@0"
I20260926 02:23:07.127841 26876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:07.149242 26876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:07.154245 26876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:07.154588 26876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:07.174280 26876 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.253:35993
I20260926 02:23:07.174310 26942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.253:35993 every 8 connection(s)
I20260926 02:23:07.175899 26876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
I20260926 02:23:07.179458 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26876
I20260926 02:23:07.179668 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-1/wal/instance
I20260926 02:23:07.180151 13267 external_mini_cluster.cc:1428] Setting key 2c6dcb9ff069df4dd7d9b344a87f09ee
I20260926 02:23:07.180510 26943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:07.188524 26943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.244.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.252@KRBTEST.COM" created.
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:07.213073 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.192389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.12.244.254:39979)
0926 02:23:07.193086 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.193095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.194127 (+ 1032us) server_negotiation.cc:407] Connection header received
0926 02:23:07.196314 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.196323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.196434 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.196574 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.198404 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.199757 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.201668 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.202227 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.202350 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.206376 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.206382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.206385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.206429 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.209491 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.210470 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.210475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.210481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.210622 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.211109 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.211115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.211119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.211428 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.211620 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.212417 (+ 797us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.212677 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":336,"threads_started":1}
I20260926 02:23:07.214123 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.193049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39979 (local address 127.0.0.1:49510)
0926 02:23:07.193911 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.193969 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.195992 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.196807 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.196816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.197526 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.198159 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.198178 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.199982 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.199986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.201447 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.201463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.202464 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.202469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.202695 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.203589 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.203617 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.206072 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.209758 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.209770 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.209793 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.210314 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.210749 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.210754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.210758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.210964 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.211575 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.211584 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.212012 (+ 428us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.212336 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.212901 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":373,"threads_started":1}
I20260926 02:23:07.216209 26943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab.
Entry for principal kudu/127.12.244.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab.
I20260926 02:23:07.234210 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.217067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36636)
0926 02:23:07.217278 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.217298 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.217423 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.218475 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.218479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.218962 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.219169 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.219187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.220517 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.220522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.221388 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.221399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.222242 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.222247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.222484 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.223212 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.223230 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.227802 (+ 4572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.231271 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.231276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.231280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.231733 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.232182 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.232187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.232191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.232388 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.233165 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.233170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.233328 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.233468 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.233784 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":3}
I20260926 02:23:07.234274 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.217260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.12.244.253:35993)
0926 02:23:07.218051 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.218057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.218083 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:23:07.218128 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.218134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.218329 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.218490 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.219334 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.220378 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.221524 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.222096 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.222232 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.228092 (+ 5860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.228115 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.228131 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.228171 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.231049 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.231871 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.231881 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.231885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.232049 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.232500 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.232508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.232511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.233006 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.233231 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.233635 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.233897 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":638,"thread_start_us":497,"threads_started":1}
I20260926 02:23:07.235818 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:44429
--webserver_interface=127.12.244.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab
--principal=kudu/127.12.244.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39979,127.12.244.253:35993,127.12.244.252:44429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:23:07.237018 26943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:23:07.238214 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.237750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59926)
0926 02:23:07.237938 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.238047 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:23:07.238471 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.240218 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.239824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59936)
0926 02:23:07.240004 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.240095 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 02:23:07.240425 26943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.289754 26943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 1
I20260926 02:23:07.291018 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.290538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59944)
0926 02:23:07.290754 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.290860 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 02:23:07.292196 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.291762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59958)
0926 02:23:07.291961 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.292061 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:23:07.292400 26943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.352402 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } attempt: 5
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
I20260926 02:23:07.360597 26943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 2
I20260926 02:23:07.361768 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.361240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59962)
0926 02:23:07.361471 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.361563 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:23:07.363013 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.362557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59966)
0926 02:23:07.362720 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.362825 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 02:23:07.363312 26943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.366950 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.353381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.12.244.253:35993)
0926 02:23:07.353595 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.353601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.353625 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:23:07.353722 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.353728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.353799 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.353907 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.354705 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.355469 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.356630 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.357091 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.357234 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.360810 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.360817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.360822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.360862 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.364148 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.365027 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.365033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.365035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.365180 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.365592 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.365596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.365598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.365880 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.365990 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.366545 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.366737 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:23:07.367383 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.353190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36644)
0926 02:23:07.353393 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.353439 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.353582 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.353947 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.353955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.354356 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.354531 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.354540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.355615 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.355621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.356484 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.356495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.357250 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.357256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.357442 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.358123 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.358142 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.360540 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.364330 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.364336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.364340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.364891 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.365299 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.365304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.365308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.365484 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.366018 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.366026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.366249 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.366443 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.366869 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 02:23:07.368659 26869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:23:07.369882 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.369387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59976)
0926 02:23:07.369595 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.369709 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 02:23:07.370852 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.370476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59982)
0926 02:23:07.370622 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.370720 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:23:07.371069 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.422411 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 1
I20260926 02:23:07.424070 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.423346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:59994)
0926 02:23:07.423633 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.423796 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 02:23:07.425436 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.424920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60002)
0926 02:23:07.425128 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.425256 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:23:07.425704 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.454586 26943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 3
I20260926 02:23:07.455926 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.455430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60018)
0926 02:23:07.455644 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.455757 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:23:07.457226 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.456680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60034)
0926 02:23:07.456964 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.457075 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260926 02:23:07.457515 26943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.493968 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 2
I20260926 02:23:07.495354 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.494658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60040)
0926 02:23:07.494924 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.495120 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:23:07.496619 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.496141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60044)
0926 02:23:07.496331 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.496444 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:23:07.496899 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
W20260926 02:23:07.590971 26951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:07.591400 26951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:07.591583 26951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:07.599169 26951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:07.599341 26951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:07.599452 26951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:07.599556 26951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:07.599644 26951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:07.600761 26943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 4
I20260926 02:23:07.601907 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.601409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60054)
0926 02:23:07.601656 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.601754 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 02:23:07.603044 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.602657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60064)
0926 02:23:07.602828 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.602916 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 02:23:07.603149 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 3
W20260926 02:23:07.603384 26943 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.604239 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.603757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60070)
0926 02:23:07.603954 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.604054 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:23:07.605374 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.604932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60078)
0926 02:23:07.605144 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.605238 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 02:23:07.605594 26869 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44429: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44429: connect: Connection refused (error 111)
I20260926 02:23:07.611274 26951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39979,127.12.244.253:35993,127.12.244.252:44429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab
--principal=kudu/127.12.244.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:44429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.244.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:07.612679 26951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:07.613998 26951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:07.627017 26957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:07.626963 26959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:07.627079 26956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:07.627445 26951 server_base.cc:1034] running on GCE node
I20260926 02:23:07.628551 26951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:07.630158 26951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:07.631479 26951 hybrid_clock.cc:648] HybridClock initialized: now 1790389387631351 us; error 68 us; skew 500 ppm
I20260926 02:23:07.631922 26951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:07.635499 26951 init.cc:377] Logged in from keytab as kudu/127.12.244.252@KRBTEST.COM (short username kudu)
I20260926 02:23:07.637080 26951 webserver.cc:460] Webserver started at http://127.12.244.252:42219/ using document root <none> and password file <none>
I20260926 02:23:07.637851 26951 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:07.638005 26951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:07.638326 26951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:07.641651 26951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data/instance:
uuid: "8709f2f45409400090f59027ebae2145"
format_stamp: "Formatted at 2026-09-26 02:23:07 on dist-test-slave-qfmw"
server_key: "0da73c3a9c361f8189a7714a26bbd964"
server_key_iv: "7b8d77c0079a4390ac367dd9568c7f70"
server_key_version: "encryptionkey@0"
I20260926 02:23:07.642577 26951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/wal/instance:
uuid: "8709f2f45409400090f59027ebae2145"
format_stamp: "Formatted at 2026-09-26 02:23:07 on dist-test-slave-qfmw"
server_key: "0da73c3a9c361f8189a7714a26bbd964"
server_key_iv: "7b8d77c0079a4390ac367dd9568c7f70"
server_key_version: "encryptionkey@0"
I20260926 02:23:07.649437 26951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260926 02:23:07.653312 26966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:07.654984 26951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:23:07.655243 26951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/wal
uuid: "8709f2f45409400090f59027ebae2145"
format_stamp: "Formatted at 2026-09-26 02:23:07 on dist-test-slave-qfmw"
server_key: "0da73c3a9c361f8189a7714a26bbd964"
server_key_iv: "7b8d77c0079a4390ac367dd9568c7f70"
server_key_version: "encryptionkey@0"
I20260926 02:23:07.655475 26951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:07.676946 26951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:07.681416 26951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:07.681730 26951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:07.701409 26951 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.252:44429
I20260926 02:23:07.701447 27017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.252:44429 every 8 connection(s)
I20260926 02:23:07.702950 26951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
I20260926 02:23:07.707168 27018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:07.710206 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 26951
I20260926 02:23:07.710399 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790389169147784-13267-0/minicluster-data/master-2/wal/instance
I20260926 02:23:07.710994 13267 external_mini_cluster.cc:1428] Setting key 278d1610b61c35aba38d5b600c91f34e
I20260926 02:23:07.716248 27018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:07.736066 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.718922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.12.244.254:39979)
0926 02:23:07.720307 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.720311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.720327 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:23:07.720378 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.720382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.720438 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.720575 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.721759 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.722431 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.724100 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.724893 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.725068 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.728795 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.728805 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.728810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.728857 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.732329 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.733330 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.733336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.733339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.733515 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.733997 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.734002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.734007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.734302 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.734430 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.735243 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.735485 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1283,"thread_start_us":1137,"threads_started":1}
I20260926 02:23:07.737334 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.718863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39979 (local address 127.0.0.1:49512)
0926 02:23:07.719677 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.719729 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.719892 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.720654 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.720661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.721312 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.721571 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.721580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.722607 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.722620 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.723904 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.723923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.725153 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.725159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.725288 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.725952 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.725977 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.728537 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.732506 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.732521 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.732536 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.733191 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.733670 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.733681 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.733684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.733858 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.734442 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.734451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.734757 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.735064 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.735771 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":54,"spinlock_wait_cycles":21376,"thread_start_us":347,"threads_started":1}
I20260926 02:23:07.737353 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.713283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.12.244.252:44429)
0926 02:23:07.714183 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.714197 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.714244 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:23:07.716081 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.716116 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.716593 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.717046 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.718628 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.720764 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.722197 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.722928 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.723199 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.728170 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.728179 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.728194 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.728229 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.732026 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.733330 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.733340 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.733343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.733588 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.733997 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.734000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.734002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.734712 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.735004 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.735357 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.735873 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":231,"server-negotiator.queue_time_us":648,"thread_start_us":404,"threads_started":1}
I20260926 02:23:07.740044 27018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:07.755466 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.740843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:35993 (local address 127.0.0.1:36660)
0926 02:23:07.741174 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.741193 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.741332 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.741658 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.741666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.742022 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.742176 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.742184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.743503 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.743510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.744667 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.744680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.745685 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.745691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.745906 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.746636 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.746654 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.748973 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.752562 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.752569 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.752573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.753163 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.753591 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.753599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.753605 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.753794 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.754337 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.754346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.754532 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.754709 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.755043 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260926 02:23:07.755563 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.741146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.12.244.253:35993)
0926 02:23:07.741344 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.741350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.741368 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:23:07.741436 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.741439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.741496 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.741621 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.742336 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.743370 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.744814 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.745478 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.745673 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.749208 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.749215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.749223 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.749276 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.752398 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.753318 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.753322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.753324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.753468 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.753922 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.753926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.753928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.754214 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.754324 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.754860 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.755071 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:23:07.756901 27018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:23:07.759001 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.740815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.12.244.254:39979)
0926 02:23:07.741019 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.741024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.741171 (+ 147us) server_negotiation.cc:407] Connection header received
0926 02:23:07.741366 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.741373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.741472 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.741545 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.743058 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.744046 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.745662 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.746286 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.746388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.752058 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.752064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.752067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.752105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.755415 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.756571 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.756579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.756585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.756736 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.757246 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.757250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.757253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.757576 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.757918 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.758280 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.758546 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
I20260926 02:23:07.772431 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.757855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60094)
0926 02:23:07.758063 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.758079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.758216 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.758628 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.758633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.759275 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.759480 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.759519 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.760770 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.760778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.762142 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.762162 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.762949 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.762955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.763240 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.763903 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.763920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.766496 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.769739 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.769746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.769751 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.770199 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.770594 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.770601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.770608 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.770809 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.771399 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.771404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.771577 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.771711 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.771957 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":3}
I20260926 02:23:07.772464 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.757761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60094 (local address 127.12.244.252:44429)
0926 02:23:07.757975 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.757982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.758147 (+ 165us) server_negotiation.cc:407] Connection header received
0926 02:23:07.758341 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.758351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.758408 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.758612 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.759680 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.760653 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.762301 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.762791 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.762938 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.766723 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.766729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.766732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.766763 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.769590 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.770335 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.770339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.770341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.770463 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.770923 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.770926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.770928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.771273 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.771418 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.771827 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.772174 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":87}
I20260926 02:23:07.778066 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.760908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.12.244.253:35993)
0926 02:23:07.761083 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.761088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.761110 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:23:07.761300 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.761305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.761365 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.761448 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.762986 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.764100 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.765793 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.766368 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.766485 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.771507 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.771512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.771516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.771550 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.775208 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.775987 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.775993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.775996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.776165 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.776565 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.776571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.776575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.776928 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.777185 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.777486 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.777732 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:23:07.778827 27018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: Bootstrap starting.
I20260926 02:23:07.782711 27018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:07.782809 26869 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 4
I20260926 02:23:07.784091 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:23:07.784314 27018 log.cc:824] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:07.788123 27018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: No bootstrap required, opened a new log
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389386, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
I20260926 02:23:07.800061 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.783414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60098)
0926 02:23:07.783622 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.783643 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.783762 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.784208 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.784212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.784687 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.784920 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.784936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.786257 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.786263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.787402 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.787420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.788233 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.788241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.788432 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.789100 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.789121 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.791472 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.796202 (+ 4730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.796211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.796218 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.796892 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.797458 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.797466 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.797472 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.797671 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.798596 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.798604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.798826 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.798989 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.799436 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 02:23:07.801659 27021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.783692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.12.244.252:44429)
0926 02:23:07.783877 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.783884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.783911 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:23:07.783969 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.783975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.784059 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.784285 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.785119 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.786103 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.787589 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.788055 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.788204 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.791741 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.791754 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.791759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.791804 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.796000 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.797091 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.797099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.797106 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.797307 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.797838 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.797846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.797851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.798393 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.800574 (+ 2181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.800748 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.801066 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:23:07.807504 27018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.808013 27018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:07.808171 27018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8709f2f45409400090f59027ebae2145, State: Initialized, Role: FOLLOWER
I20260926 02:23:07.809085 26869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21: Bootstrap starting.
I20260926 02:23:07.814231 26869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:07.815748 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.794688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.12.244.254:39979)
0926 02:23:07.794876 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.794882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.794901 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:23:07.794957 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.794964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.795040 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.795205 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.798422 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.800030 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.802987 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.803700 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.803805 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.807945 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.807953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.807957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.808008 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.812165 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.813119 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.813125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.813129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.813290 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.813826 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.813831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.813834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.814236 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.814506 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.815080 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.815356 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:23:07.815874 26869 log.cc:824] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:07.815862 27018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.821779 27034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:07.822305 27034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:23:07.824322 27018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:07.827657 26869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21: No bootstrap required, opened a new log
I20260926 02:23:07.830111 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.795335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.12.244.252:44429)
0926 02:23:07.802859 (+ 7524us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.802868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.809209 (+ 6341us) server_negotiation.cc:407] Connection header received
0926 02:23:07.809495 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.809501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.809591 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.809677 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.811991 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.813264 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.816526 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.817204 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.817315 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.822003 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.822011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.822016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.822066 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.826064 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.827253 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.827259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.827262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.827462 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.827998 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.828004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.828007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.828443 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.828741 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.829177 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.829432 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7363,"thread_start_us":7759,"threads_started":1}
I20260926 02:23:07.838143 26869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.838543 26869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:07.838670 26869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4407a95db22f4774bee77a2aadbd8e21, State: Initialized, Role: FOLLOWER
I20260926 02:23:07.839447 26869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.840225 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.794904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.12.244.253:35993)
0926 02:23:07.795488 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.795496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.811478 (+ 15982us) server_negotiation.cc:407] Connection header received
0926 02:23:07.812092 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.812100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.812210 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.812286 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.814502 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.815626 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.818852 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.819835 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.819967 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.831369 (+ 11402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.831379 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.831387 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.831458 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.836084 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.837093 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.837101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.837106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.837285 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.837841 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.837848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.837854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.838342 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.838649 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.839409 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.839706 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441}
I20260926 02:23:07.842010 26869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:07.840790 27046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:07.842957 27046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 02:23:07.854117 27045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:23:07.854346 27045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:23:07.846535 26951 master_runner.cc:186] Error getting master registration for 127.12.244.254:39979: OK, instance_id { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" instance_seqno: 1790389386637958 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:23:07.863976 26951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:23:07.870445 27058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:23:07.870754 27058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:23:07.860752 26801 master_runner.cc:186] Error getting master registration for 127.12.244.253:35993: OK, instance_id { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" instance_seqno: 1790389387168542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:23:07.872345 26801 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 02:23:07.876989 26822 tablet.cc:2369] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:07.893723 27046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:23:07.894131 27046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.896063 27046 leader_election.cc:288] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a138a2a3de4c4f26892fcade34338707 (127.12.244.253:35993), 8709f2f45409400090f59027ebae2145 (127.12.244.252:44429)
I20260926 02:23:07.897183 26993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4407a95db22f4774bee77a2aadbd8e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8709f2f45409400090f59027ebae2145" is_pre_election: true
I20260926 02:23:07.897919 26993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4407a95db22f4774bee77a2aadbd8e21 in term 0.
I20260926 02:23:07.898662 26818 leader_election.cc:302] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4407a95db22f4774bee77a2aadbd8e21, 8709f2f45409400090f59027ebae2145; no voters:
I20260926 02:23:07.899165 27046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:23:07.899339 27046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:23:07.899448 27046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:07.902704 26943 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } attempt: 5
I20260926 02:23:07.905344 27046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.907401 26993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4407a95db22f4774bee77a2aadbd8e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8709f2f45409400090f59027ebae2145"
I20260926 02:23:07.907660 26993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:07.911671 26993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4407a95db22f4774bee77a2aadbd8e21 in term 1.
I20260926 02:23:07.912683 26818 leader_election.cc:302] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4407a95db22f4774bee77a2aadbd8e21, 8709f2f45409400090f59027ebae2145; no voters:
I20260926 02:23:07.917147 27046 leader_election.cc:288] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [CANDIDATE]: Term 1 election: Requested vote from peers a138a2a3de4c4f26892fcade34338707 (127.12.244.253:35993), 8709f2f45409400090f59027ebae2145 (127.12.244.252:44429)
Sep 26 02:23:07 dist-test-slave-qfmw krb5kdc[26788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389387, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
I20260926 02:23:07.918321 27046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:07.919183 27046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [term 1 LEADER]: Becoming Leader. State: Replica: 4407a95db22f4774bee77a2aadbd8e21, State: Running, Role: LEADER
I20260926 02:23:07.921959 27046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.932219 27032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.904182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.12.244.252:44429)
0926 02:23:07.904585 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:07.904591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:07.904620 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:23:07.904681 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:07.904686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:07.904784 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:07.904887 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:07.906536 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.907739 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.914090 (+ 6351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.914856 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.914990 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.919548 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:07.919556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:07.919562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:07.919620 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:07.923585 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.924707 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.924713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.924717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.924937 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:07.929418 (+ 4481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:07.929425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:07.929429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:07.929950 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:07.930116 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:07.931274 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0926 02:23:07.931575 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260926 02:23:07.932960 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:07.903393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44429 (local address 127.0.0.1:60116)
0926 02:23:07.903813 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:07.903837 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:07.903963 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:07.905448 (+ 1485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:07.905453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:07.906096 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:07.906341 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.906355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.910130 (+ 3775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.910137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.911687 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:07.911707 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:07.915261 (+ 3554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:07.915269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:07.915512 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:07.916262 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:07.916289 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:07.919281 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:07.923806 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.923812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.923816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.924469 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.925095 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:07.925101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:07.925105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:07.929244 (+ 4139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:07.930192 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:07.930198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:07.930381 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 02:23:07.930565 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 02:23:07.932473 (+ 1908us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
W20260926 02:23:07.933995 26972 tablet.cc:2369] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:07.940382 26943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707: Bootstrap starting.
I20260926 02:23:07.945696 27060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4407a95db22f4774bee77a2aadbd8e21. Latest consensus state: current_term: 1 leader_uuid: "4407a95db22f4774bee77a2aadbd8e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:07.946187 27060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:07.953565 26943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:07.958441 26943 log.cc:824] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:07.961249 27062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:23:07.963915 27062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:23:07.972188 26943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707: No bootstrap required, opened a new log
I20260926 02:23:07.975553 26993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [term 1 FOLLOWER]: Refusing update from remote peer 4407a95db22f4774bee77a2aadbd8e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:07.976805 27060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:07.990060 26943 raft_consensus.cc:348] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:07.990669 26943 raft_consensus.cc:374] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:07.990826 26943 raft_consensus.cc:729] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a138a2a3de4c4f26892fcade34338707, State: Initialized, Role: FOLLOWER
I20260926 02:23:07.993939 26943 consensus_queue.cc:260] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } }
I20260926 02:23:08.005167 27034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4407a95db22f4774bee77a2aadbd8e21. Latest consensus state: current_term: 1 leader_uuid: "4407a95db22f4774bee77a2aadbd8e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:08.005554 27034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:23:08.006093 27067 sys_catalog.cc:455] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:08.009368 27067 sys_catalog.cc:458] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:23:08.007982 26943 sys_catalog.cc:564] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:08.007915 26918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4407a95db22f4774bee77a2aadbd8e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a138a2a3de4c4f26892fcade34338707" is_pre_election: true
I20260926 02:23:08.008373 26916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:08.011409 27064 mvcc.cc:205] Tried to move back new op lower bound from 7333434933135773696 to 7333434932950986752. Current Snapshot: MvccSnapshot[applied={T|T < 7333434933135773696}]
I20260926 02:23:08.008399 26917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4407a95db22f4774bee77a2aadbd8e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a138a2a3de4c4f26892fcade34338707"
I20260926 02:23:08.014041 26916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [term 1 FOLLOWER]: Refusing update from remote peer 4407a95db22f4774bee77a2aadbd8e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:08.014739 27065 mvcc.cc:205] Tried to move back new op lower bound from 7333434933135773696 to 7333434932950986752. Current Snapshot: MvccSnapshot[applied={T|T < 7333434933135773696}]
I20260926 02:23:08.015157 27060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:08.035410 27060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4407a95db22f4774bee77a2aadbd8e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:08.035851 27060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:08.040988 27034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4407a95db22f4774bee77a2aadbd8e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:08.041294 27034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:23:08.043883 27062 catalog_manager.cc:1347] Generated new cluster ID: 63c6525aac274eb5b967671b54e6105a
I20260926 02:23:08.044034 27062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:23:08.048465 27075 sys_catalog.cc:455] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4407a95db22f4774bee77a2aadbd8e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:08.048943 27075 sys_catalog.cc:458] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:23:08.049288 27067 sys_catalog.cc:455] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4407a95db22f4774bee77a2aadbd8e21. Latest consensus state: current_term: 1 leader_uuid: "4407a95db22f4774bee77a2aadbd8e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:08.049531 27067 sys_catalog.cc:458] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:23:08.061380 27060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4407a95db22f4774bee77a2aadbd8e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4407a95db22f4774bee77a2aadbd8e21" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39979 } } peers { permanent_uuid: "a138a2a3de4c4f26892fcade34338707" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 35993 } } peers { permanent_uuid: "8709f2f45409400090f59027ebae2145" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44429 } } }
I20260926 02:23:08.061728 27060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:08.069695 27062 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:23:08.071954 27062 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:23:08.095999 27062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4407a95db22f4774bee77a2aadbd8e21: Generated new TSK 0
I20260926 02:23:08.097455 27062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:23:08.113353 27085 catalog_manager.cc:1259] Loaded cluster ID: 63c6525aac274eb5b967671b54e6105a
I20260926 02:23:08.113478 27085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707: loading cluster ID for follower catalog manager: success
I20260926 02:23:08.117736 27085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707: acquiring CA information for follower catalog manager: success
I20260926 02:23:08.132436 27085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a138a2a3de4c4f26892fcade34338707: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:23:08.856559 27045 catalog_manager.cc:1259] Loaded cluster ID: 63c6525aac274eb5b967671b54e6105a
I20260926 02:23:08.856715 27045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: loading cluster ID for follower catalog manager: success
I20260926 02:23:08.859373 27045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: acquiring CA information for follower catalog manager: success
I20260926 02:23:08.861387 27045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8709f2f45409400090f59027ebae2145: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:23:09.267036 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26801
I20260926 02:23:09.268719 26864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:09.381314 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26801
I20260926 02:23:09.402508 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26876
I20260926 02:23:09.403667 26938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:09.512015 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26876
I20260926 02:23:09.533011 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 26951
I20260926 02:23:09.534195 27013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:09.643378 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 26951
2026-09-26T02:23:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4851 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:23:09 dist-test-slave-qfmw krb5kdc[27099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:23:09 dist-test-slave-qfmw krb5kdc[27099](info): set up 2 sockets
Sep 26 02:23:09 dist-test-slave-qfmw krb5kdc[27099](info): commencing operation
krb5kdc: starting...
W20260926 02:23:10.793969 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:23:10 dist-test-slave-qfmw krb5kdc[27099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:23:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:23:11.043043 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:35051
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:46383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:11.399966 27112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:11.400291 27112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:11.400393 27112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:11.408090 27112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:11.408221 27112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:11.408268 27112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:11.408327 27112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:11.408381 27112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:11.418871 27112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:46383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:11.420320 27112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:11.421726 27112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:11.435228 27117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:11.435201 27120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:11.435169 27118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:11.435890 27112 server_base.cc:1034] running on GCE node
I20260926 02:23:11.437054 27112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:11.438596 27112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:11.439812 27112 hybrid_clock.cc:648] HybridClock initialized: now 1790389391439756 us; error 48 us; skew 500 ppm
I20260926 02:23:11.440203 27112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:11 dist-test-slave-qfmw krb5kdc[27099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389391, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:11.443849 27112 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:23:11.445575 27112 webserver.cc:460] Webserver started at http://127.12.244.254:40921/ using document root <none> and password file <none>
I20260926 02:23:11.446362 27112 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:11.446512 27112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:11.446872 27112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:11.450191 27112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "c02c411190fc4f38b6577d48402d1355"
format_stamp: "Formatted at 2026-09-26 02:23:11 on dist-test-slave-qfmw"
server_key: "ab2bfeb3929d7cb45b8ecdddab816fd3"
server_key_iv: "6cf9d6ebd9f6dba1553230508a9f9170"
server_key_version: "encryptionkey@0"
I20260926 02:23:11.451040 27112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "c02c411190fc4f38b6577d48402d1355"
format_stamp: "Formatted at 2026-09-26 02:23:11 on dist-test-slave-qfmw"
server_key: "ab2bfeb3929d7cb45b8ecdddab816fd3"
server_key_iv: "6cf9d6ebd9f6dba1553230508a9f9170"
server_key_version: "encryptionkey@0"
I20260926 02:23:11.456823 27112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 02:23:11.460798 27127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:11.462303 27112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:23:11.462502 27112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "c02c411190fc4f38b6577d48402d1355"
format_stamp: "Formatted at 2026-09-26 02:23:11 on dist-test-slave-qfmw"
server_key: "ab2bfeb3929d7cb45b8ecdddab816fd3"
server_key_iv: "6cf9d6ebd9f6dba1553230508a9f9170"
server_key_version: "encryptionkey@0"
I20260926 02:23:11.462743 27112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:11.479852 27112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:11.484905 27112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:11.485236 27112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:11.505221 27112 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:35051
I20260926 02:23:11.505245 27179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:35051 every 8 connection(s)
I20260926 02:23:11.506793 27112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:23:11.507994 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27112
I20260926 02:23:11.508247 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:23:11.508919 13267 external_mini_cluster.cc:1428] Setting key 8101d499b8b7569e71a4e7f781ab45f9
I20260926 02:23:11.511592 27180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:23:11 dist-test-slave-qfmw krb5kdc[27099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:11.526659 27180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355: Bootstrap starting.
I20260926 02:23:11.530241 27180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:11.531487 27180 log.cc:824] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:11.535365 27180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355: No bootstrap required, opened a new log
I20260926 02:23:11.535677 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:11.511197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.12.244.254:35051)
0926 02:23:11.512376 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:11.512389 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:11.512435 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:23:11.514223 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:11.514248 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:11.514708 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:11.515158 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:11.517234 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:11.518749 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:11.520760 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:11.521512 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:11.521881 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:11.526942 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:11.526952 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:11.526971 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:11.527019 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:11.530372 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:11.531408 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:11.531417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:11.531424 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:11.531651 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:11.532212 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:11.532218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:11.532223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:11.533137 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:11.533406 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:11.534004 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 02:23:11.534456 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":583,"threads_started":1}
I20260926 02:23:11.539992 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:23:11.545099 27180 raft_consensus.cc:348] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02c411190fc4f38b6577d48402d1355" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35051 } }
I20260926 02:23:11.545527 27180 raft_consensus.cc:374] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:11.545609 27180 raft_consensus.cc:729] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c02c411190fc4f38b6577d48402d1355, State: Initialized, Role: FOLLOWER
I20260926 02:23:11.546159 27180 consensus_queue.cc:260] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02c411190fc4f38b6577d48402d1355" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35051 } }
I20260926 02:23:11.546371 27180 raft_consensus.cc:388] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:23:11.546465 27180 raft_consensus.cc:482] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:23:11.546594 27180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:11.548905 27180 raft_consensus.cc:504] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02c411190fc4f38b6577d48402d1355" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35051 } }
I20260926 02:23:11.549427 27180 leader_election.cc:302] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c02c411190fc4f38b6577d48402d1355; no voters:
I20260926 02:23:11.550012 27180 leader_election.cc:288] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:23:11.550206 27185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:11.550927 27185 raft_consensus.cc:686] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [term 1 LEADER]: Becoming Leader. State: Replica: c02c411190fc4f38b6577d48402d1355, State: Running, Role: LEADER
I20260926 02:23:11.551533 27185 consensus_queue.cc:237] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02c411190fc4f38b6577d48402d1355" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35051 } }
I20260926 02:23:11.552376 27180 sys_catalog.cc:564] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:11.554564 27187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c02c411190fc4f38b6577d48402d1355. Latest consensus state: current_term: 1 leader_uuid: "c02c411190fc4f38b6577d48402d1355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02c411190fc4f38b6577d48402d1355" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35051 } } }
I20260926 02:23:11.554558 27186 sys_catalog.cc:455] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c02c411190fc4f38b6577d48402d1355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c02c411190fc4f38b6577d48402d1355" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35051 } } }
I20260926 02:23:11.555051 27187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:11.555231 27186 sys_catalog.cc:458] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:11.557293 27191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:23:11.562878 27191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:23:11.573261 27191 catalog_manager.cc:1347] Generated new cluster ID: 50eca7ffccc84a7cbcb84e4f5568d733
I20260926 02:23:11.573410 27191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:23:11.602720 27191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:23:11.604285 27191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:23:11.610949 27191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c02c411190fc4f38b6577d48402d1355: Generated new TSK 0
I20260926 02:23:11.612219 27191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:23:11.653556 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27112
I20260926 02:23:11.654852 27175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:11.752833 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27112
2026-09-26T02:23:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2105 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:23:11 dist-test-slave-qfmw krb5kdc[27208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:23:11 dist-test-slave-qfmw krb5kdc[27208](info): set up 2 sockets
Sep 26 02:23:11 dist-test-slave-qfmw krb5kdc[27208](info): commencing operation
krb5kdc: starting...
W20260926 02:23:12.878237 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:23:13 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:23:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:23:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:23:13.135684 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45941
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:13.489954 27221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:13.490279 27221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:13.490440 27221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:13.498080 27221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:13.498212 27221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:13.498298 27221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:13.498389 27221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:13.498510 27221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:13.509235 27221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:13.510696 27221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:13.512164 27221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:13.525497 27226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:13.525457 27229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:13.525543 27227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:13.526976 27221 server_base.cc:1034] running on GCE node
I20260926 02:23:13.527987 27221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:13.529716 27221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:13.530963 27221 hybrid_clock.cc:648] HybridClock initialized: now 1790389393530912 us; error 51 us; skew 500 ppm
I20260926 02:23:13.531391 27221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:13 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389393, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:13.534839 27221 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:23:13.536625 27221 webserver.cc:460] Webserver started at http://127.12.244.254:37223/ using document root <none> and password file <none>
I20260926 02:23:13.537370 27221 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:13.537524 27221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:13.537943 27221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:13.541251 27221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "ffd06c898b964614a469fddd135d9926"
format_stamp: "Formatted at 2026-09-26 02:23:13 on dist-test-slave-qfmw"
server_key: "f7c43430e9f5f6449b5e2bb3007aed38"
server_key_iv: "b7ca1ef1f760c06221f5ec5bd3cd8a6e"
server_key_version: "encryptionkey@0"
I20260926 02:23:13.542088 27221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "ffd06c898b964614a469fddd135d9926"
format_stamp: "Formatted at 2026-09-26 02:23:13 on dist-test-slave-qfmw"
server_key: "f7c43430e9f5f6449b5e2bb3007aed38"
server_key_iv: "b7ca1ef1f760c06221f5ec5bd3cd8a6e"
server_key_version: "encryptionkey@0"
I20260926 02:23:13.547894 27221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 02:23:13.551970 27236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:13.553629 27221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:23:13.553845 27221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "ffd06c898b964614a469fddd135d9926"
format_stamp: "Formatted at 2026-09-26 02:23:13 on dist-test-slave-qfmw"
server_key: "f7c43430e9f5f6449b5e2bb3007aed38"
server_key_iv: "b7ca1ef1f760c06221f5ec5bd3cd8a6e"
server_key_version: "encryptionkey@0"
I20260926 02:23:13.554092 27221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:13.579458 27221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:13.584412 27221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:13.584790 27221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:13.605486 27221 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:45941
I20260926 02:23:13.605585 27287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:45941 every 8 connection(s)
I20260926 02:23:13.607065 27221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:23:13.611337 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27221
I20260926 02:23:13.611433 27288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:13.611603 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:23:13.612516 13267 external_mini_cluster.cc:1428] Setting key ddee1e1ac3dfdc6eb17401992a50c712
I20260926 02:23:13.627352 27288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Bootstrap starting.
Sep 26 02:23:13 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:13.631852 27288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:13.633760 27288 log.cc:824] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:13.637916 27288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: No bootstrap required, opened a new log
I20260926 02:23:13.642160 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:13.614826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.12.244.254:45941)
0926 02:23:13.615783 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:13.615797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:13.615842 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:23:13.618392 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:13.618422 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:13.618932 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:13.619411 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:13.621292 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:13.622810 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:13.624933 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:13.625757 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:13.625925 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:13.631428 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:13.631454 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:13.631480 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:13.631550 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:13.636235 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:13.637245 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:13.637254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:13.637263 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:13.637499 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:13.638228 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:13.638234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:13.638237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:13.639084 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:13.639463 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:13.639947 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 02:23:13.640510 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":479,"threads_started":1}
I20260926 02:23:13.649063 27288 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:13.649658 27288 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:13.649786 27288 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd06c898b964614a469fddd135d9926, State: Initialized, Role: FOLLOWER
I20260926 02:23:13.650478 27288 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:13.650748 27288 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:23:13.650913 27288 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:23:13.651134 27288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:13.653475 27288 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:13.654069 27288 leader_election.cc:302] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd06c898b964614a469fddd135d9926; no voters:
I20260926 02:23:13.654723 27288 leader_election.cc:288] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:23:13.654937 27293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:13.656293 27293 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 LEADER]: Becoming Leader. State: Replica: ffd06c898b964614a469fddd135d9926, State: Running, Role: LEADER
I20260926 02:23:13.657012 27293 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:13.657253 27288 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:13.660732 27295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd06c898b964614a469fddd135d9926. Latest consensus state: current_term: 1 leader_uuid: "ffd06c898b964614a469fddd135d9926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } } }
I20260926 02:23:13.660656 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd06c898b964614a469fddd135d9926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } } }
I20260926 02:23:13.661280 27295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:13.661296 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:13.668604 27301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:23:13.674813 27301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:23:13.686369 27301 catalog_manager.cc:1347] Generated new cluster ID: 16ae0ca24ced4beb9348bd4adbbc2803
I20260926 02:23:13.686623 27301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:23:13.721381 27301 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:23:13.723675 27301 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:23:13.737635 27301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Generated new TSK 0
I20260926 02:23:13.738992 27301 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:23:13.815946 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45941
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:14.190701 27315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:14.191012 27315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:14.191191 27315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:14.198680 27315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:14.198822 27315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:14.198946 27315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:23:14.209376 27315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:14.210857 27315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:14.212242 27315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:14.226601 27320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:14.226532 27323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:14.226577 27321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:14.227887 27315 server_base.cc:1034] running on GCE node
I20260926 02:23:14.228680 27315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:14.229755 27315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:14.231014 27315 hybrid_clock.cc:648] HybridClock initialized: now 1790389394230966 us; error 51 us; skew 500 ppm
I20260926 02:23:14.231436 27315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:14 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389394, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:14.234891 27315 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:23:14.236685 27315 webserver.cc:460] Webserver started at http://127.12.244.193:41213/ using document root <none> and password file <none>
I20260926 02:23:14.237532 27315 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:14.237690 27315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:14.238013 27315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:14.241384 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "284a97c2a29e4ac9bce9973c4cea63eb"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "670689d5cda547516c3b36db7fa11143"
server_key_iv: "9a81ec73a028ed240c0d9a245ad70e8d"
server_key_version: "encryptionkey@0"
I20260926 02:23:14.242280 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "284a97c2a29e4ac9bce9973c4cea63eb"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "670689d5cda547516c3b36db7fa11143"
server_key_iv: "9a81ec73a028ed240c0d9a245ad70e8d"
server_key_version: "encryptionkey@0"
I20260926 02:23:14.248381 27315 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:23:14.252345 27330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:14.253886 27315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:23:14.254109 27315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "284a97c2a29e4ac9bce9973c4cea63eb"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "670689d5cda547516c3b36db7fa11143"
server_key_iv: "9a81ec73a028ed240c0d9a245ad70e8d"
server_key_version: "encryptionkey@0"
I20260926 02:23:14.254341 27315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:14.275488 27315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:14.280108 27315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:14.280471 27315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:14.281615 27315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:14.283071 27315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:14.283238 27315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 02:23:14.283398 27315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:14.283492 27315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:14.320503 27315 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:33957
I20260926 02:23:14.320607 27443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:33957 every 8 connection(s)
I20260926 02:23:14.321955 27315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:23:14.322639 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27315
I20260926 02:23:14.322963 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:23:14.323658 13267 external_mini_cluster.cc:1428] Setting key 4d2ca3ffe78f6d7b46111cf1558b3b69
Sep 26 02:23:14 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389394, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:14.348515 27446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:14.325260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:60253 (local address 127.12.244.254:45941)
0926 02:23:14.325762 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:14.325768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:14.327113 (+ 1345us) server_negotiation.cc:407] Connection header received
0926 02:23:14.330127 (+ 3014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:14.330131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:14.330212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:14.330316 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:14.332866 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.333674 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.335717 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.336467 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.336635 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:14.340550 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:14.340560 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:14.340569 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:14.340629 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:14.344419 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:14.345393 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:14.345402 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:14.345408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:14.345573 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:14.346250 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:14.346257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:14.346262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:14.346706 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:14.346901 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:14.347808 (+ 907us) server_negotiation.cc:299] Negotiation successful
0926 02:23:14.348083 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260926 02:23:14.349282 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:14.325987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:60253)
0926 02:23:14.326890 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:14.326944 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:14.329740 (+ 2796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:14.330659 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:14.330675 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:14.331665 (+ 990us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:14.332611 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:14.332643 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.333871 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.333879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:14.335550 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:14.335563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.336718 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.336723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:14.336943 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:14.337695 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:14.337725 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:14.340254 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:14.344665 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:14.344674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:14.344691 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:14.345226 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:14.345741 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:14.345745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:14.345747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:14.346097 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:14.346864 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:14.346870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:14.347338 (+ 468us) client_negotiation.cc:769] Sending connection context
0926 02:23:14.347635 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 02:23:14.348086 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":316,"threads_started":1}
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:23:14.351235 27444 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45941
I20260926 02:23:14.351590 27444 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:14.352826 27444 heartbeater.cc:511] Master 127.12.244.254:45941 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:23:14.355480 27253 ts_manager.cc:194] Registered new tserver with Master: 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957)
I20260926 02:23:14.357088 27253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:60253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:23:14.382877 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45941
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:14.738504 27450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:14.738821 27450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:14.738978 27450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:14.746631 27450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:14.746759 27450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:14.746938 27450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:23:14.757588 27450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:14.759248 27450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:14.760775 27450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:14.774922 27456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:14.774977 27458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:14.775010 27455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:14.780943 27450 server_base.cc:1034] running on GCE node
I20260926 02:23:14.781700 27450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:14.782773 27450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:14.784044 27450 hybrid_clock.cc:648] HybridClock initialized: now 1790389394783951 us; error 87 us; skew 500 ppm
I20260926 02:23:14.784461 27450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:14 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389394, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:14.787878 27450 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:23:14.789618 27450 webserver.cc:460] Webserver started at http://127.12.244.194:40483/ using document root <none> and password file <none>
I20260926 02:23:14.790396 27450 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:14.790551 27450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:14.790879 27450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:14.794390 27450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "f1a244344e47423fae3d291ad756daf5"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "2a6acfb2f36aab7aa036103845542c18"
server_key_iv: "7713bd670168b256983383ac7232e49e"
server_key_version: "encryptionkey@0"
I20260926 02:23:14.795655 27450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "f1a244344e47423fae3d291ad756daf5"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "2a6acfb2f36aab7aa036103845542c18"
server_key_iv: "7713bd670168b256983383ac7232e49e"
server_key_version: "encryptionkey@0"
I20260926 02:23:14.804973 27450 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.008s
I20260926 02:23:14.808962 27465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:14.810519 27450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:23:14.810756 27450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "f1a244344e47423fae3d291ad756daf5"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "2a6acfb2f36aab7aa036103845542c18"
server_key_iv: "7713bd670168b256983383ac7232e49e"
server_key_version: "encryptionkey@0"
I20260926 02:23:14.811002 27450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:14.829878 27450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:14.834487 27450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:14.834820 27450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:14.835987 27450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:14.837455 27450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:14.837569 27450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:14.837735 27450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:14.837807 27450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:14.873346 27450 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:33057
I20260926 02:23:14.873409 27579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:33057 every 8 connection(s)
I20260926 02:23:14.874718 27450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:23:14.878134 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27450
I20260926 02:23:14.878336 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:23:14.878793 13267 external_mini_cluster.cc:1428] Setting key 0040e598d94081508a1c3a126f7e0632
Sep 26 02:23:14 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389394, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:14.898041 27582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:14.877522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:60073 (local address 127.12.244.254:45941)
0926 02:23:14.878031 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:14.878037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:14.879242 (+ 1205us) server_negotiation.cc:407] Connection header received
0926 02:23:14.881385 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:14.881391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:14.881480 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:14.881622 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:14.883478 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.884686 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.886394 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.887223 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.887369 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:14.891552 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:14.891559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:14.891562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:14.891600 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:14.894460 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:14.895473 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:14.895478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:14.895480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:14.895624 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:14.896128 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:14.896132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:14.896134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:14.896415 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:14.896597 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:14.897377 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 02:23:14.897639 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":238,"threads_started":1}
I20260926 02:23:14.899220 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:14.878109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:60073)
0926 02:23:14.879028 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:14.879076 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:14.881084 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:14.881908 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:14.881918 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:14.882522 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:14.883274 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:14.883299 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.884871 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.884876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:14.886224 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:14.886239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:14.887446 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:14.887451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:14.887674 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:14.888455 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:14.888485 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:14.891241 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:14.894670 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:14.894684 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:14.894708 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:14.895302 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:14.895752 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:14.895759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:14.895767 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:14.895977 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:14.896568 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:14.896577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:14.896982 (+ 405us) client_negotiation.cc:769] Sending connection context
0926 02:23:14.897283 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 02:23:14.897887 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":407,"threads_started":1}
I20260926 02:23:14.900902 27580 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45941
I20260926 02:23:14.901293 27580 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
I20260926 02:23:14.902347 27580 heartbeater.cc:511] Master 127.12.244.254:45941 requested a full tablet report, sending...
I20260926 02:23:14.904069 27253 ts_manager.cc:194] Registered new tserver with Master: f1a244344e47423fae3d291ad756daf5 (127.12.244.194:33057)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
I20260926 02:23:14.905032 27253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:60073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:23:14.934710 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45941
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:15.287005 27586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:15.287453 27586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:15.287664 27586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:15.297011 27586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:15.297149 27586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:15.297384 27586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:23:15.308175 27586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:15.309612 27586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:15.311210 27586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:15.325519 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:15.326193 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:15.326567 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:15.326882 27586 server_base.cc:1034] running on GCE node
I20260926 02:23:15.327634 27586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:15.328702 27586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:15.329970 27586 hybrid_clock.cc:648] HybridClock initialized: now 1790389395329866 us; error 72 us; skew 500 ppm
I20260926 02:23:15.330364 27586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:15 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389395, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:15.334585 27586 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:23:15.336386 27586 webserver.cc:460] Webserver started at http://127.12.244.195:40993/ using document root <none> and password file <none>
I20260926 02:23:15.337200 27586 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:15.337365 27586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:15.337728 27586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:15.341248 27586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "569f349128cd4c819ed58392f0de77b0"
format_stamp: "Formatted at 2026-09-26 02:23:15 on dist-test-slave-qfmw"
server_key: "4e490c2636219de62379f8d8e954ef33"
server_key_iv: "426b553d20bf23d0b67619a6637bc881"
server_key_version: "encryptionkey@0"
I20260926 02:23:15.342358 27586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "569f349128cd4c819ed58392f0de77b0"
format_stamp: "Formatted at 2026-09-26 02:23:15 on dist-test-slave-qfmw"
server_key: "4e490c2636219de62379f8d8e954ef33"
server_key_iv: "426b553d20bf23d0b67619a6637bc881"
server_key_version: "encryptionkey@0"
I20260926 02:23:15.348542 27586 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 02:23:15.352552 27601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:15.354127 27586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:23:15.354334 27586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "569f349128cd4c819ed58392f0de77b0"
format_stamp: "Formatted at 2026-09-26 02:23:15 on dist-test-slave-qfmw"
server_key: "4e490c2636219de62379f8d8e954ef33"
server_key_iv: "426b553d20bf23d0b67619a6637bc881"
server_key_version: "encryptionkey@0"
I20260926 02:23:15.354563 27586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:15.360949 27444 heartbeater.cc:503] Master 127.12.244.254:45941 was elected leader, sending a full tablet report...
I20260926 02:23:15.384572 27586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:15.389374 27586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:15.389649 27586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:15.390602 27586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:15.392165 27586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:15.392246 27586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:15.392340 27586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:15.392387 27586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:15.427668 27586 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:35777
I20260926 02:23:15.427778 27714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:35777 every 8 connection(s)
I20260926 02:23:15.429049 27586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:23:15.430240 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27586
I20260926 02:23:15.430438 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:23:15.431027 13267 external_mini_cluster.cc:1428] Setting key 6463260c1c0bb7cc0953d2f2c37ec519
Sep 26 02:23:15 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389395, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:15.451153 27717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.432171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:37301 (local address 127.12.244.254:45941)
0926 02:23:15.432673 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.432680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.433944 (+ 1264us) server_negotiation.cc:407] Connection header received
0926 02:23:15.436056 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.436062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.436166 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.436298 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:15.437917 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.438765 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.440444 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.440860 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.441019 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.444881 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:15.444886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:15.444890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:15.444934 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:15.447842 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.448686 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.448691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.448693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.448819 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.449287 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.449296 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.449298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.449575 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:15.449745 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.450492 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.450723 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260926 02:23:15.452180 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.432879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:37301)
0926 02:23:15.433795 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:15.433839 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:15.435770 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:15.436562 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:15.436572 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:15.437129 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:15.437736 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.437754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.438940 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.438944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.440285 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.440298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.441125 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.441132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.441390 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.442135 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:15.442159 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:15.444604 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:15.448013 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:15.448024 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:15.448046 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:15.448541 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:15.448935 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:15.448940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:15.448943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:15.449162 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:15.449707 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:15.449716 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:15.450109 (+ 393us) client_negotiation.cc:769] Sending connection context
0926 02:23:15.450383 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 02:23:15.450970 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":372,"threads_started":1}
I20260926 02:23:15.453605 27715 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45941
I20260926 02:23:15.453933 27715 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:15.454710 27715 heartbeater.cc:511] Master 127.12.244.254:45941 requested a full tablet report, sending...
I20260926 02:23:15.456418 27253 ts_manager.cc:194] Registered new tserver with Master: 569f349128cd4c819ed58392f0de77b0 (127.12.244.195:35777)
I20260926 02:23:15.457384 27253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:37301
I20260926 02:23:15.459762 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:23:15.478317 27717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.464453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.12.244.254:45941)
0926 02:23:15.464700 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.464708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.465082 (+ 374us) server_negotiation.cc:407] Connection header received
0926 02:23:15.465303 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.465307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.465385 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.465493 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:15.466967 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.467847 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.469278 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.469682 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.469820 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.472457 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:15.472462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:15.472466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:15.472504 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:15.475303 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.476113 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.476119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.476125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.476278 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.476700 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.476705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.476708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.477063 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:15.477345 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.477700 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.477968 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:23:15.482751 27253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:23:15.489878 27253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:23:15.525885 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.503347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.12.244.194:33057)
0926 02:23:15.515764 (+ 12417us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.515778 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.517787 (+ 2009us) server_negotiation.cc:407] Connection header received
0926 02:23:15.517979 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.517992 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.518271 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.518477 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:15.519417 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.520756 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.522855 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.524593 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.524792 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.524995 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.525318 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.525497 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12147,"thread_start_us":12445,"threads_started":1}
I20260926 02:23:15.526067 27729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.502854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:33957 (local address 127.0.0.1:34072)
0926 02:23:15.513046 (+ 10192us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:15.513115 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:15.513335 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:15.517881 (+ 4546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:15.517891 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:15.517931 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:15.518491 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.518510 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.520564 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.520582 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.523028 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.523043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.525173 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.525179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.525367 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.525401 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:23:15.525494 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 02:23:15.525644 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9917,"mutex_wait_us":2,"thread_start_us":15190,"threads_started":1}
I20260926 02:23:15.526319 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.503224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.12.244.193:33957)
0926 02:23:15.516589 (+ 13365us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.516598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.516638 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:23:15.516777 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.516790 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.517255 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.517512 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:15.518736 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.520388 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.523302 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.524816 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.525008 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.525175 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.525786 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.525982 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13188,"thread_start_us":13010,"threads_started":1}
I20260926 02:23:15.526464 27732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.503133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:33057 (local address 127.0.0.1:40136)
0926 02:23:15.517613 (+ 14480us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:15.517643 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:15.517791 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:15.518739 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:15.518744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:15.518766 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:15.519197 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.519214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.520901 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.520907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.522645 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.522663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.524817 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.524824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.525043 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.525120 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 02:23:15.525270 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 02:23:15.525492 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14327,"mutex_wait_us":3,"thread_start_us":9464,"threads_started":1}
I20260926 02:23:15.527525 27728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.505794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:35777 (local address 127.0.0.1:44050)
0926 02:23:15.518406 (+ 12612us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:15.518433 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:15.518663 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:15.519686 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:15.519691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:15.519717 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:15.520078 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.520090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.522083 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.522089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.524392 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.524407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.526696 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.526702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.526852 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.526875 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:23:15.526998 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 02:23:15.527162 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12464,"mutex_wait_us":3,"thread_start_us":11493,"threads_started":1}
I20260926 02:23:15.527582 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.506485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.12.244.195:35777)
0926 02:23:15.517094 (+ 10609us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.517103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.518550 (+ 1447us) server_negotiation.cc:407] Connection header received
0926 02:23:15.518862 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.518871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.519154 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.519338 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:15.520256 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.521845 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.524583 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.526073 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.526279 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.526950 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.527033 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.527252 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10417,"thread_start_us":10278,"threads_started":1}
I20260926 02:23:15.530927 27649 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef1ce6166ca4319a0c7f1999480e9f6 (DEFAULT_TABLE table=test-table [id=49519cc026734e8eb28c6424395d342f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:15.530942 27514 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef1ce6166ca4319a0c7f1999480e9f6 (DEFAULT_TABLE table=test-table [id=49519cc026734e8eb28c6424395d342f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:15.531409 27379 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef1ce6166ca4319a0c7f1999480e9f6 (DEFAULT_TABLE table=test-table [id=49519cc026734e8eb28c6424395d342f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:15.532550 27649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef1ce6166ca4319a0c7f1999480e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:15.532544 27514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef1ce6166ca4319a0c7f1999480e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:15.532734 27379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef1ce6166ca4319a0c7f1999480e9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:15.544902 27734 tablet_bootstrap.cc:492] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Bootstrap starting.
I20260926 02:23:15.547773 27736 tablet_bootstrap.cc:492] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Bootstrap starting.
I20260926 02:23:15.548959 27735 tablet_bootstrap.cc:492] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Bootstrap starting.
I20260926 02:23:15.549229 27734 tablet_bootstrap.cc:654] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:15.551759 27736 tablet_bootstrap.cc:654] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:15.551578 27734 log.cc:824] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:15.552819 27735 tablet_bootstrap.cc:654] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:15.553097 27736 log.cc:824] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:15.554070 27735 log.cc:824] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:15.555280 27734 tablet_bootstrap.cc:492] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: No bootstrap required, opened a new log
I20260926 02:23:15.555596 27734 ts_tablet_manager.cc:1397] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260926 02:23:15.556445 27736 tablet_bootstrap.cc:492] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: No bootstrap required, opened a new log
I20260926 02:23:15.556730 27736 ts_tablet_manager.cc:1397] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 02:23:15.556879 27735 tablet_bootstrap.cc:492] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: No bootstrap required, opened a new log
I20260926 02:23:15.557139 27735 ts_tablet_manager.cc:1397] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:23:15.565786 27735 raft_consensus.cc:348] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.566251 27735 raft_consensus.cc:374] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:15.566401 27735 raft_consensus.cc:729] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284a97c2a29e4ac9bce9973c4cea63eb, State: Initialized, Role: FOLLOWER
I20260926 02:23:15.567143 27735 consensus_queue.cc:260] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.567757 27734 raft_consensus.cc:348] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.567757 27736 raft_consensus.cc:348] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.568256 27736 raft_consensus.cc:374] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:15.568318 27734 raft_consensus.cc:374] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:15.568406 27736 raft_consensus.cc:729] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a244344e47423fae3d291ad756daf5, State: Initialized, Role: FOLLOWER
I20260926 02:23:15.568476 27734 raft_consensus.cc:729] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569f349128cd4c819ed58392f0de77b0, State: Initialized, Role: FOLLOWER
I20260926 02:23:15.569119 27736 consensus_queue.cc:260] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.569173 27734 consensus_queue.cc:260] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.570344 27715 heartbeater.cc:503] Master 127.12.244.254:45941 was elected leader, sending a full tablet report...
I20260926 02:23:15.571076 27734 ts_tablet_manager.cc:1428] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 02:23:15.571527 27580 heartbeater.cc:503] Master 127.12.244.254:45941 was elected leader, sending a full tablet report...
I20260926 02:23:15.572852 27736 ts_tablet_manager.cc:1428] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260926 02:23:15.574651 27735 ts_tablet_manager.cc:1428] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
W20260926 02:23:15.575966 27445 tablet.cc:2369] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:15.579331 27742 raft_consensus.cc:482] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:23:15.579609 27742 raft_consensus.cc:504] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.581246 27742 leader_election.cc:288] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957), 569f349128cd4c819ed58392f0de77b0 (127.12.244.195:35777)
I20260926 02:23:15.588441 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.581655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:58959 (local address 127.12.244.193:33957)
0926 02:23:15.581864 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.581869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.583003 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 02:23:15.583253 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.583257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.583318 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.583418 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:15.583937 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.584645 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.586370 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.587478 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.587633 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.587719 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.588042 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.588180 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:23:15.588490 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.581865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:54497 (local address 127.12.244.195:35777)
0926 02:23:15.582103 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.582110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.582520 (+ 410us) server_negotiation.cc:407] Connection header received
0926 02:23:15.582791 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.582797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.582869 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.582970 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:15.583510 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.584268 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.586323 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.587422 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.587597 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.587672 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.588099 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.588231 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:23:15.588650 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.581707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:35777 (local address 127.12.244.194:54497)
0926 02:23:15.582413 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:15.582429 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:15.582639 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:15.582986 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:15.582991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:15.583017 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:15.583366 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.583375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.584402 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.584410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.586124 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.586140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.587817 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.587822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.587937 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.587960 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:23:15.588066 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:23:15.588210 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":79,"thread_start_us":405,"threads_started":1}
I20260926 02:23:15.589272 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.581720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:33957 (local address 127.12.244.194:58959)
0926 02:23:15.582886 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:15.582901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:15.583056 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:15.583430 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:15.583435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:15.583456 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:15.583788 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.583796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.584781 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.584788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.586212 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:15.586224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.587706 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.587713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:15.587876 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.587901 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:23:15.588007 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:23:15.588149 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":24,"spinlock_wait_cycles":22784,"thread_start_us":799,"threads_started":1}
I20260926 02:23:15.589852 27670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef1ce6166ca4319a0c7f1999480e9f6" candidate_uuid: "f1a244344e47423fae3d291ad756daf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569f349128cd4c819ed58392f0de77b0" is_pre_election: true
I20260926 02:23:15.590363 27399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef1ce6166ca4319a0c7f1999480e9f6" candidate_uuid: "f1a244344e47423fae3d291ad756daf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" is_pre_election: true
I20260926 02:23:15.590523 27670 raft_consensus.cc:2454] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a244344e47423fae3d291ad756daf5 in term 0.
I20260926 02:23:15.590932 27399 raft_consensus.cc:2454] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a244344e47423fae3d291ad756daf5 in term 0.
I20260926 02:23:15.591303 27466 leader_election.cc:302] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569f349128cd4c819ed58392f0de77b0, f1a244344e47423fae3d291ad756daf5; no voters:
I20260926 02:23:15.591754 27742 raft_consensus.cc:2781] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:23:15.591917 27742 raft_consensus.cc:482] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:23:15.592027 27742 raft_consensus.cc:3037] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:15.594290 27742 raft_consensus.cc:504] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.594908 27742 leader_election.cc:288] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [CANDIDATE]: Term 1 election: Requested vote from peers 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957), 569f349128cd4c819ed58392f0de77b0 (127.12.244.195:35777)
I20260926 02:23:15.595598 27670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef1ce6166ca4319a0c7f1999480e9f6" candidate_uuid: "f1a244344e47423fae3d291ad756daf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "569f349128cd4c819ed58392f0de77b0"
I20260926 02:23:15.595647 27399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef1ce6166ca4319a0c7f1999480e9f6" candidate_uuid: "f1a244344e47423fae3d291ad756daf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284a97c2a29e4ac9bce9973c4cea63eb"
I20260926 02:23:15.595973 27670 raft_consensus.cc:3037] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:15.595994 27399 raft_consensus.cc:3037] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:15.599572 27399 raft_consensus.cc:2454] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a244344e47423fae3d291ad756daf5 in term 1.
I20260926 02:23:15.599572 27670 raft_consensus.cc:2454] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a244344e47423fae3d291ad756daf5 in term 1.
I20260926 02:23:15.600210 27466 leader_election.cc:302] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 284a97c2a29e4ac9bce9973c4cea63eb, 569f349128cd4c819ed58392f0de77b0, f1a244344e47423fae3d291ad756daf5; no voters:
I20260926 02:23:15.600611 27742 raft_consensus.cc:2781] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:15.601155 27742 raft_consensus.cc:686] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 1 LEADER]: Becoming Leader. State: Replica: f1a244344e47423fae3d291ad756daf5, State: Running, Role: LEADER
I20260926 02:23:15.601711 27742 consensus_queue.cc:237] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } }
I20260926 02:23:15.607241 27252 catalog_manager.cc:5495] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a244344e47423fae3d291ad756daf5 (127.12.244.194). New cstate: current_term: 1 leader_uuid: "f1a244344e47423fae3d291ad756daf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } health_report { overall_health: UNKNOWN } } }
W20260926 02:23:15.626864 27581 tablet.cc:2369] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:15.636823 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.631275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.12.244.194:33057)
0926 02:23:15.631460 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.631464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.631613 (+ 149us) server_negotiation.cc:407] Connection header received
0926 02:23:15.631821 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.631825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.631903 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.632039 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:23:15.632444 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.633216 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.634664 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.635050 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.635246 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.635716 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:23:15.636266 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:23:15.636405 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.636535 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.636610 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:23:15.646097 27399 raft_consensus.cc:1261] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Refusing update from remote peer f1a244344e47423fae3d291ad756daf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:15.646098 27670 raft_consensus.cc:1261] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 1 FOLLOWER]: Refusing update from remote peer f1a244344e47423fae3d291ad756daf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:15.647195 27745 consensus_queue.cc:1035] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:15.647612 27742 consensus_queue.cc:1035] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:15.660655 27748 mvcc.cc:205] Tried to move back new op lower bound from 7333434964555374592 to 7333434964389543936. Current Snapshot: MvccSnapshot[applied={T|T < 7333434964555374592}]
I20260926 02:23:15.682302 27251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:15.682690 27251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:15.686916 27251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Sending DeleteTablet for 3 replicas of tablet 1ef1ce6166ca4319a0c7f1999480e9f6
I20260926 02:23:15.688246 27649 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef1ce6166ca4319a0c7f1999480e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:15 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:44050
I20260926 02:23:15.688244 27379 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef1ce6166ca4319a0c7f1999480e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:15 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:34072
I20260926 02:23:15.689424 27758 tablet_replica.cc:330] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: stopping tablet replica
I20260926 02:23:15.689852 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27315
I20260926 02:23:15.689963 27758 raft_consensus.cc:2229] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:15.689030 27514 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef1ce6166ca4319a0c7f1999480e9f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:15 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:40136
I20260926 02:23:15.690510 27758 raft_consensus.cc:2258] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:15.691296 27759 tablet_replica.cc:330] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: stopping tablet replica
I20260926 02:23:15.691795 27759 raft_consensus.cc:2229] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:15.692351 27759 raft_consensus.cc:2258] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:15.694773 27758 ts_tablet_manager.cc:1905] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:15.700045 27759 ts_tablet_manager.cc:1905] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:15.701161 27760 tablet_replica.cc:330] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: stopping tablet replica
I20260926 02:23:15.701956 27760 raft_consensus.cc:2229] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:23:15.702319 27758 ts_tablet_manager.cc:1918] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:15.702520 27758 log.cc:1196] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/1ef1ce6166ca4319a0c7f1999480e9f6
I20260926 02:23:15.702971 27760 raft_consensus.cc:2258] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:15.703133 27758 ts_tablet_manager.cc:1939] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 284a97c2a29e4ac9bce9973c4cea63eb: Deleting consensus metadata
I20260926 02:23:15.705202 27238 catalog_manager.cc:4841] TS 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957): tablet 1ef1ce6166ca4319a0c7f1999480e9f6 (table test-table [id=49519cc026734e8eb28c6424395d342f]) successfully deleted
I20260926 02:23:15.705518 27760 ts_tablet_manager.cc:1905] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:15.710309 27759 ts_tablet_manager.cc:1918] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:15.710649 27759 log.cc:1196] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/1ef1ce6166ca4319a0c7f1999480e9f6
I20260926 02:23:15.711436 27759 ts_tablet_manager.cc:1939] T 1ef1ce6166ca4319a0c7f1999480e9f6 P 569f349128cd4c819ed58392f0de77b0: Deleting consensus metadata
I20260926 02:23:15.713538 27237 catalog_manager.cc:4841] TS 569f349128cd4c819ed58392f0de77b0 (127.12.244.195:35777): tablet 1ef1ce6166ca4319a0c7f1999480e9f6 (table test-table [id=49519cc026734e8eb28c6424395d342f]) successfully deleted
I20260926 02:23:15.713541 27760 ts_tablet_manager.cc:1918] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:15.714133 27760 log.cc:1196] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/1ef1ce6166ca4319a0c7f1999480e9f6
I20260926 02:23:15.714669 27760 ts_tablet_manager.cc:1939] T 1ef1ce6166ca4319a0c7f1999480e9f6 P f1a244344e47423fae3d291ad756daf5: Deleting consensus metadata
I20260926 02:23:15.716460 27240 catalog_manager.cc:4841] TS f1a244344e47423fae3d291ad756daf5 (127.12.244.194:33057): tablet 1ef1ce6166ca4319a0c7f1999480e9f6 (table test-table [id=49519cc026734e8eb28c6424395d342f]) successfully deleted
Sep 26 02:23:15 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:23:15.725859 27730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.691557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.12.244.193:33957)
0926 02:23:15.691756 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.691763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.705472 (+ 13709us) server_negotiation.cc:407] Connection header received
0926 02:23:15.705788 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.705794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.705884 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.705964 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:15.708156 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.709338 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.711839 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.712556 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.712672 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.718291 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:15.718315 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:15.718335 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:15.718389 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:15.722460 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.723492 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.723508 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.723513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.723703 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.724159 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.724165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.724168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.724668 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:15.725006 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.725201 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.725454 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:23:15.726645 27439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:15.819164 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27315
I20260926 02:23:15.845957 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27450
Sep 26 02:23:15 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:23:15.872246 27731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:15.846758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.12.244.194:33057)
0926 02:23:15.846951 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:15.846956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:15.847009 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:23:15.847269 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:15.847273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:15.847339 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:15.847431 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:15.849360 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.850214 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.852593 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:15.853026 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:15.853154 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:15.863811 (+ 10657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:15.863847 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:15.863871 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:15.863952 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:15.868543 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.869487 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.869498 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.869502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.869696 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:15.870240 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:15.870281 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:15.870285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:15.870866 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:15.871276 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:15.871518 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 02:23:15.871768 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:23:15.873068 27575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:15.991990 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27450
I20260926 02:23:16.007722 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27586
Sep 26 02:23:16 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:23:16.023168 27733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:16.008466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.12.244.195:35777)
0926 02:23:16.008679 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:16.008684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:16.008703 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:23:16.008763 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:16.008766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:16.008838 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:16.008922 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:16.010080 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:16.010833 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:16.012375 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:16.012779 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:16.012880 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:16.017405 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:16.017421 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:16.017438 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:16.017479 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:16.020289 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:16.021006 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:16.021014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:16.021016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:16.021178 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:16.021526 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:16.021529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:16.021531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:16.021896 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:16.022137 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:16.022534 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 02:23:16.022798 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:23:16.023910 27710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:16.116499 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27586
I20260926 02:23:16.133498 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27221
I20260926 02:23:16.134685 27283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:16.246510 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27221
I20260926 02:23:16.268472 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45941
--webserver_interface=127.12.244.254
--webserver_port=37223
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:16.619527 27766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:16.619925 27766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:16.620110 27766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:16.627943 27766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:16.628123 27766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:16.628247 27766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:16.628346 27766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:16.628435 27766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:16.639282 27766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=37223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:16.640729 27766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:16.642184 27766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:16.666385 27772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:16.666435 27775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:16.666579 27773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:16.667531 27766 server_base.cc:1034] running on GCE node
I20260926 02:23:16.668644 27766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:16.670497 27766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:16.671810 27766 hybrid_clock.cc:648] HybridClock initialized: now 1790389396671728 us; error 76 us; skew 500 ppm
I20260926 02:23:16.672350 27766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:16 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389396, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:16.676609 27766 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:23:16.678712 27766 webserver.cc:460] Webserver started at http://127.12.244.254:37223/ using document root <none> and password file <none>
I20260926 02:23:16.679683 27766 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:16.679833 27766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:16.686604 27766 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:23:16.690286 27782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:16.692371 27766 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 02:23:16.692591 27766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "ffd06c898b964614a469fddd135d9926"
format_stamp: "Formatted at 2026-09-26 02:23:13 on dist-test-slave-qfmw"
server_key: "f7c43430e9f5f6449b5e2bb3007aed38"
server_key_iv: "b7ca1ef1f760c06221f5ec5bd3cd8a6e"
server_key_version: "encryptionkey@0"
I20260926 02:23:16.693799 27766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:16.711670 27766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:16.717154 27766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:16.717417 27766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:16.738660 27766 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:45941
I20260926 02:23:16.738701 27834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:45941 every 8 connection(s)
I20260926 02:23:16.740466 27766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:23:16.743144 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27766
I20260926 02:23:16.744179 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:33957
--local_ip_for_outbound_sockets=127.12.244.193
--tserver_master_addrs=127.12.244.254:45941
--webserver_port=41213
--webserver_interface=127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:23:16.751384 27835 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:23:16.755820 27835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Bootstrap starting.
I20260926 02:23:16.777036 27835 log.cc:824] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:16.790136 27835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:23:16.790736 27835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Bootstrap complete.
I20260926 02:23:16.800673 27835 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:16.801699 27835 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd06c898b964614a469fddd135d9926, State: Initialized, Role: FOLLOWER
I20260926 02:23:16.802527 27835 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:16.802837 27835 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:23:16.802973 27835 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:23:16.803200 27835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:23:16.807379 27835 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:16.807987 27835 leader_election.cc:302] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd06c898b964614a469fddd135d9926; no voters:
I20260926 02:23:16.808672 27835 leader_election.cc:288] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:23:16.808821 27839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:23:16.810112 27839 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 LEADER]: Becoming Leader. State: Replica: ffd06c898b964614a469fddd135d9926, State: Running, Role: LEADER
I20260926 02:23:16.810685 27839 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:16.811244 27835 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:16.813282 27841 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd06c898b964614a469fddd135d9926. Latest consensus state: current_term: 2 leader_uuid: "ffd06c898b964614a469fddd135d9926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } } }
I20260926 02:23:16.813695 27841 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:16.814114 27840 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffd06c898b964614a469fddd135d9926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } } }
I20260926 02:23:16.814396 27840 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:16.814642 27845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:23:16.836583 27845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:23:16.838636 27845 catalog_manager.cc:1259] Loaded cluster ID: 16ae0ca24ced4beb9348bd4adbbc2803
I20260926 02:23:16.838790 27845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:23:16.844465 27845 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 02:23:16.873072 27845 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 02:23:16.885998 27845 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:23:16.887893 27845 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Loaded TSK: 0
I20260926 02:23:16.889591 27845 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:23:17.128981 27837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:17.129360 27837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:17.129521 27837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:17.137302 27837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:17.137478 27837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:17.137666 27837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:23:17.148386 27837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:33957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=41213
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:17.149809 27837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:17.151309 27837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:17.167848 27862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:17.167716 27865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:17.167716 27863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:17.168937 27837 server_base.cc:1034] running on GCE node
I20260926 02:23:17.169836 27837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:17.171221 27837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:17.172485 27837 hybrid_clock.cc:648] HybridClock initialized: now 1790389397172428 us; error 44 us; skew 500 ppm
I20260926 02:23:17.172912 27837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:17 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389397, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:17.183168 27837 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:23:17.185231 27837 webserver.cc:460] Webserver started at http://127.12.244.193:41213/ using document root <none> and password file <none>
I20260926 02:23:17.186250 27837 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:17.186416 27837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:17.194555 27837 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:23:17.198489 27872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:17.200555 27837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 02:23:17.200766 27837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "284a97c2a29e4ac9bce9973c4cea63eb"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "670689d5cda547516c3b36db7fa11143"
server_key_iv: "9a81ec73a028ed240c0d9a245ad70e8d"
server_key_version: "encryptionkey@0"
I20260926 02:23:17.202189 27837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:17.249889 27837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:17.254217 27837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:17.254458 27837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:17.255523 27837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:17.256915 27837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:17.257018 27837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:17.257093 27837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:17.257148 27837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:17.292018 27837 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:33957
I20260926 02:23:17.292109 27985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:33957 every 8 connection(s)
I20260926 02:23:17.293594 27837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:23:17.303215 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27837
I20260926 02:23:17.303838 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:33057
--local_ip_for_outbound_sockets=127.12.244.194
--tserver_master_addrs=127.12.244.254:45941
--webserver_port=40483
--webserver_interface=127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:23:17 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389397, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:17.322034 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:17.296605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:38751 (local address 127.12.244.254:45941)
0926 02:23:17.297176 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:17.297188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:17.298004 (+ 816us) server_negotiation.cc:407] Connection header received
0926 02:23:17.301051 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:17.301073 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:17.301700 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:17.302416 (+ 716us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:17.304374 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.305671 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.307450 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.308248 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.308500 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:17.313358 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:17.313367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:17.313385 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:17.313433 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:17.316688 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:17.317778 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:17.317786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:17.317792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:17.317976 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:17.318572 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:17.318576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:17.318578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:17.319180 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:17.319360 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:17.320455 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 02:23:17.320822 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260926 02:23:17.322299 27989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:17.296907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:38751)
0926 02:23:17.297784 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:17.297837 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:17.300241 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:17.302560 (+ 2319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:17.302573 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:17.303403 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:17.304108 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:17.304142 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.305857 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.305863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:17.307285 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:17.307301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.308550 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.308558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:17.308854 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:17.309597 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:17.309621 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:17.313015 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:17.316875 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:17.316888 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:17.316909 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:17.317613 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:17.318114 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:17.318122 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:17.318129 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:17.318396 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:17.319348 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:17.319360 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:17.319852 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 02:23:17.320194 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 02:23:17.320844 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":350,"threads_started":1}
I20260926 02:23:17.325888 27986 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45941
I20260926 02:23:17.326272 27986 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:17.327169 27986 heartbeater.cc:511] Master 127.12.244.254:45941 requested a full tablet report, sending...
I20260926 02:23:17.329877 27800 ts_manager.cc:194] Registered new tserver with Master: 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957)
I20260926 02:23:17.332077 27800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:38751
W20260926 02:23:17.653384 27990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:17.653726 27990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:17.653833 27990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:17.662560 27990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:17.662685 27990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:17.662812 27990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:23:17.674201 27990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:33057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=40483
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:17.675681 27990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:17.677120 27990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:17.692615 27995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:17.692550 27998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:17.692584 27996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:17.693449 27990 server_base.cc:1034] running on GCE node
I20260926 02:23:17.694223 27990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:17.695326 27990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:17.696574 27990 hybrid_clock.cc:648] HybridClock initialized: now 1790389397696515 us; error 55 us; skew 500 ppm
I20260926 02:23:17.696933 27990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:17 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389397, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:17.700399 27990 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:23:17.702078 27990 webserver.cc:460] Webserver started at http://127.12.244.194:40483/ using document root <none> and password file <none>
I20260926 02:23:17.702822 27990 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:17.702988 27990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:17.708981 27990 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:23:17.712189 28005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:17.713761 27990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:23:17.713958 27990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "f1a244344e47423fae3d291ad756daf5"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "2a6acfb2f36aab7aa036103845542c18"
server_key_iv: "7713bd670168b256983383ac7232e49e"
server_key_version: "encryptionkey@0"
I20260926 02:23:17.715066 27990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:17.732379 27990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:17.736884 27990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:17.737193 27990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:17.738400 27990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:17.740300 27990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:17.740427 27990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:17.740540 27990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:17.740620 27990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:17.780416 27990 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:33057
I20260926 02:23:17.780555 28117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:33057 every 8 connection(s)
I20260926 02:23:17.781996 27990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:23:17.791011 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 27990
I20260926 02:23:17.791769 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:35777
--local_ip_for_outbound_sockets=127.12.244.195
--tserver_master_addrs=127.12.244.254:45941
--webserver_port=40993
--webserver_interface=127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:23:17 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389397, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:17.804734 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:17.784773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:43205 (local address 127.12.244.254:45941)
0926 02:23:17.784983 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:17.784989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:17.786238 (+ 1249us) server_negotiation.cc:407] Connection header received
0926 02:23:17.788084 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:17.788088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:17.788178 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:17.788273 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:17.789744 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.790448 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.793023 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.793457 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.793561 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:17.797994 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:17.797999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:17.798003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:17.798037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:17.801310 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:17.802220 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:17.802225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:17.802227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:17.802364 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:17.802832 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:17.802836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:17.802838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:17.803191 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:17.803346 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:17.804107 (+ 761us) server_negotiation.cc:299] Negotiation successful
0926 02:23:17.804365 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:23:17.805799 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:17.785323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:43205)
0926 02:23:17.786071 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:17.786114 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:17.787829 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:17.788477 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:17.788485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:17.789030 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:17.789580 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:17.789608 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.790663 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.790669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:17.792843 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:17.792862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:17.793702 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:17.793712 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:17.794042 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:17.794853 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:17.794889 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:17.797739 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:17.801518 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:17.801535 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:17.801550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:17.802061 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:17.802480 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:17.802485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:17.802489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:17.802698 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:17.803320 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:17.803329 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:17.803708 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 02:23:17.804005 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 02:23:17.804606 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":280,"threads_started":1}
I20260926 02:23:17.807555 28118 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45941
I20260926 02:23:17.807906 28118 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:17.808854 28118 heartbeater.cc:511] Master 127.12.244.254:45941 requested a full tablet report, sending...
I20260926 02:23:17.810739 27800 ts_manager.cc:194] Registered new tserver with Master: f1a244344e47423fae3d291ad756daf5 (127.12.244.194:33057)
I20260926 02:23:17.812207 27800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:43205
W20260926 02:23:18.138912 28121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:18.139287 28121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:18.139434 28121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:18.146963 28121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:18.147162 28121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:18.147305 28121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:23:18.158031 28121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:35777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=40993
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:18.159610 28121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:18.161056 28121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:18.175668 28126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:18.175608 28128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:18.175835 28130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:18.176481 28121 server_base.cc:1034] running on GCE node
I20260926 02:23:18.177156 28121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:18.178273 28121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:18.179513 28121 hybrid_clock.cc:648] HybridClock initialized: now 1790389398179469 us; error 38 us; skew 500 ppm
I20260926 02:23:18.179937 28121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:18 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389398, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:18.183696 28121 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:23:18.185478 28121 webserver.cc:460] Webserver started at http://127.12.244.195:40993/ using document root <none> and password file <none>
I20260926 02:23:18.186295 28121 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:18.186631 28121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:18.193202 28121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:23:18.196826 28137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:18.198629 28121 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:23:18.198828 28121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "569f349128cd4c819ed58392f0de77b0"
format_stamp: "Formatted at 2026-09-26 02:23:15 on dist-test-slave-qfmw"
server_key: "4e490c2636219de62379f8d8e954ef33"
server_key_iv: "426b553d20bf23d0b67619a6637bc881"
server_key_version: "encryptionkey@0"
I20260926 02:23:18.200055 28121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:18.220352 28121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:18.225252 28121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:18.225558 28121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:18.226608 28121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:18.228240 28121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:18.228374 28121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:18.228508 28121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:18.228618 28121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:18.263582 28121 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:35777
I20260926 02:23:18.263625 28249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:35777 every 8 connection(s)
I20260926 02:23:18.265132 28121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:23:18.270139 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 28121
Sep 26 02:23:18 dist-test-slave-qfmw krb5kdc[27208](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389398, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:18.286055 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.271057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.12.244.254:45941)
0926 02:23:18.271552 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.271558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.271951 (+ 393us) server_negotiation.cc:407] Connection header received
0926 02:23:18.272073 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.272077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.272157 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.272267 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:18.273735 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.274546 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.275990 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.276548 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.276690 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.279788 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:18.279794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:18.279797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:18.279832 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:18.283188 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.283899 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.283905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.283908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.284043 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.284467 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.284476 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.284478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.284767 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:18.285052 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.285439 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.285687 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":389,"thread_start_us":226,"threads_started":1}
I20260926 02:23:18.287137 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.267872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:48525 (local address 127.12.244.254:45941)
0926 02:23:18.268068 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.268073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.269313 (+ 1240us) server_negotiation.cc:407] Connection header received
0926 02:23:18.271466 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.271473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.271554 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.271635 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:18.273647 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.274421 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.276090 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.276674 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.276806 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.280514 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:18.280525 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:18.280528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:18.280571 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:18.283423 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.284294 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.284299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.284301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.284419 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.284889 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.284894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.284899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.285274 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:18.285415 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.286366 (+ 951us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.286547 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":68}
I20260926 02:23:18.288797 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.268343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:48525)
0926 02:23:18.269135 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:18.269177 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:18.271037 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:18.272198 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:18.272214 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:18.272856 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:18.273454 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.273472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.274580 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.274586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.275935 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.275947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.276913 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.276921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.277172 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.277800 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:18.277823 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:18.280239 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:18.283596 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:18.283605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:18.283630 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:18.284154 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:18.284536 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:18.284542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:18.284546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:18.284764 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:18.285493 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:18.285501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:18.285905 (+ 404us) client_negotiation.cc:769] Sending connection context
0926 02:23:18.287060 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0926 02:23:18.287697 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":315,"threads_started":1}
I20260926 02:23:18.290220 28250 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45941
I20260926 02:23:18.290566 28250 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:18.291399 28250 heartbeater.cc:511] Master 127.12.244.254:45941 requested a full tablet report, sending...
I20260926 02:23:18.293052 27800 ts_manager.cc:194] Registered new tserver with Master: 569f349128cd4c819ed58392f0de77b0 (127.12.244.195:35777)
I20260926 02:23:18.293933 27800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:48525
I20260926 02:23:18.301566 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:23:18.309835 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.302802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.12.244.254:45941)
0926 02:23:18.303317 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.303323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.303505 (+ 182us) server_negotiation.cc:407] Connection header received
0926 02:23:18.303782 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.303788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.304038 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.304212 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:23:18.304754 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.305803 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.307512 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.307963 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.308120 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.308703 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:23:18.309247 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:23:18.309410 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.309478 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.309568 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260926 02:23:18.310842 27800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:23:18.318656 27800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:23:18.336017 27986 heartbeater.cc:503] Master 127.12.244.254:45941 was elected leader, sending a full tablet report...
I20260926 02:23:18.355464 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.331904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:33057 (local address 127.0.0.1:40162)
0926 02:23:18.342980 (+ 11076us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:18.343050 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:18.343347 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:18.346643 (+ 3296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:18.347956 (+ 1313us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:18.347994 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:18.348551 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.348566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.350576 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.350590 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.352973 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.352985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.354396 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.354402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.354613 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.354672 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 02:23:18.354816 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 02:23:18.355026 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10653,"mutex_wait_us":28,"thread_start_us":14320,"threads_started":1}
I20260926 02:23:18.356664 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.332063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.12.244.194:33057)
0926 02:23:18.345611 (+ 13548us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.345624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.345663 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:23:18.345796 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.345805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.346136 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.346346 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:18.348796 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.350438 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.353155 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.354150 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.354354 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.354574 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.354874 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.356053 (+ 1179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13368,"thread_start_us":13236,"threads_started":1}
I20260926 02:23:18.357589 28260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.332089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:35777 (local address 127.0.0.1:44060)
0926 02:23:18.345969 (+ 13880us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:18.345990 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:18.346121 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:18.348433 (+ 2312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:18.348439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:18.348478 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:18.348909 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.348925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.351463 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.351470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.354204 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.354222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.356605 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.356612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.356800 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.356858 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 02:23:18.356933 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 02:23:18.357087 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13754,"mutex_wait_us":2,"thread_start_us":10279,"threads_started":1}
I20260926 02:23:18.358502 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.332680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:33957 (local address 127.0.0.1:34092)
0926 02:23:18.348259 (+ 15579us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:18.348287 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:18.348451 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:18.349538 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:18.349542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:18.349569 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:18.349918 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.349928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.352357 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.352366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.354839 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.354854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.357362 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.357369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.357598 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.357643 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:23:18.357987 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 02:23:18.358138 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15399,"thread_start_us":15204,"threads_started":1}
I20260926 02:23:18.358620 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.333379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.12.244.193:33957)
0926 02:23:18.347226 (+ 13847us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.347235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.348613 (+ 1378us) server_negotiation.cc:407] Connection header received
0926 02:23:18.348728 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.348737 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.349285 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.349522 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:18.350120 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.352156 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.355060 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.356754 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.356987 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.357189 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.358078 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.358273 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13630,"thread_start_us":13461,"threads_started":1}
I20260926 02:23:18.358860 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.332858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.12.244.195:35777)
0926 02:23:18.347745 (+ 14887us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.347755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.347792 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:23:18.347897 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.347904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.348190 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.349025 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:18.349334 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.351263 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.354491 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.356239 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.357956 (+ 1717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.358165 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.358257 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.358451 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14704,"thread_start_us":14557,"threads_started":1}
I20260926 02:23:18.360965 28053 tablet_service.cc:1467] Processing CreateTablet for tablet cbbf207443364e10a3a2415074046de9 (DEFAULT_TABLE table=test-table [id=1fca6fa3991543ccb4d9ca9441d2999d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:18.362926 27920 tablet_service.cc:1467] Processing CreateTablet for tablet cbbf207443364e10a3a2415074046de9 (DEFAULT_TABLE table=test-table [id=1fca6fa3991543ccb4d9ca9441d2999d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:18.362913 28185 tablet_service.cc:1467] Processing CreateTablet for tablet cbbf207443364e10a3a2415074046de9 (DEFAULT_TABLE table=test-table [id=1fca6fa3991543ccb4d9ca9441d2999d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:18.363833 28053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbf207443364e10a3a2415074046de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:18.364595 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbf207443364e10a3a2415074046de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:18.364612 27920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbbf207443364e10a3a2415074046de9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:18.376981 28266 tablet_bootstrap.cc:492] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Bootstrap starting.
I20260926 02:23:18.381593 28266 tablet_bootstrap.cc:654] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:18.381669 28267 tablet_bootstrap.cc:492] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Bootstrap starting.
I20260926 02:23:18.383214 28266 log.cc:824] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:18.383769 28268 tablet_bootstrap.cc:492] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Bootstrap starting.
I20260926 02:23:18.385475 28267 tablet_bootstrap.cc:654] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:18.386538 28267 log.cc:824] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:18.387495 28268 tablet_bootstrap.cc:654] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:18.387765 28266 tablet_bootstrap.cc:492] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: No bootstrap required, opened a new log
I20260926 02:23:18.388069 28266 ts_tablet_manager.cc:1397] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260926 02:23:18.389155 28268 log.cc:824] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:18.389896 28267 tablet_bootstrap.cc:492] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: No bootstrap required, opened a new log
I20260926 02:23:18.390201 28267 ts_tablet_manager.cc:1397] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:23:18.392561 28268 tablet_bootstrap.cc:492] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: No bootstrap required, opened a new log
I20260926 02:23:18.392849 28268 ts_tablet_manager.cc:1397] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:23:18.397624 28267 raft_consensus.cc:348] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.398077 28267 raft_consensus.cc:374] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:18.398190 28267 raft_consensus.cc:729] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284a97c2a29e4ac9bce9973c4cea63eb, State: Initialized, Role: FOLLOWER
I20260926 02:23:18.398903 28267 consensus_queue.cc:260] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.398917 28266 raft_consensus.cc:348] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.399415 28266 raft_consensus.cc:374] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:18.399508 28266 raft_consensus.cc:729] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a244344e47423fae3d291ad756daf5, State: Initialized, Role: FOLLOWER
I20260926 02:23:18.400461 28266 consensus_queue.cc:260] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.403551 28267 ts_tablet_manager.cc:1428] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260926 02:23:18.404691 28118 heartbeater.cc:503] Master 127.12.244.254:45941 was elected leader, sending a full tablet report...
I20260926 02:23:18.404719 28268 raft_consensus.cc:348] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.405303 28268 raft_consensus.cc:374] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:18.405400 28268 raft_consensus.cc:729] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569f349128cd4c819ed58392f0de77b0, State: Initialized, Role: FOLLOWER
I20260926 02:23:18.405557 28266 ts_tablet_manager.cc:1428] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260926 02:23:18.406180 28268 consensus_queue.cc:260] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.407660 28250 heartbeater.cc:503] Master 127.12.244.254:45941 was elected leader, sending a full tablet report...
I20260926 02:23:18.408592 28268 ts_tablet_manager.cc:1428] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20260926 02:23:18.516832 28251 tablet.cc:2369] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:23:18.534227 28119 tablet.cc:2369] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:23:18.545955 27987 tablet.cc:2369] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:18.624939 28274 raft_consensus.cc:482] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:23:18.625257 28274 raft_consensus.cc:504] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.627290 28274 leader_election.cc:288] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a244344e47423fae3d291ad756daf5 (127.12.244.194:33057), 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957)
I20260926 02:23:18.633018 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.627578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:33057 (local address 127.12.244.195:45657)
0926 02:23:18.627932 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:18.627955 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:18.628129 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:18.628526 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:18.628531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:18.628557 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:18.628778 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.628786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.629799 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.629804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.631148 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.631171 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.632264 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.632270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.632459 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.632484 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:23:18.632589 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:23:18.632732 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260926 02:23:18.633018 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.627605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:45657 (local address 127.12.244.194:33057)
0926 02:23:18.627960 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.627967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.628023 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:23:18.628328 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.628333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.628391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.628495 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:18.628925 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.629662 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.631305 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.632063 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.632281 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.632358 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.632629 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.632760 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260926 02:23:18.634366 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.627949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:47073 (local address 127.12.244.193:33957)
0926 02:23:18.628228 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.628234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.628847 (+ 613us) server_negotiation.cc:407] Connection header received
0926 02:23:18.629074 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.629079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.629150 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.629253 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:18.629864 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.630607 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.632433 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.633452 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.633639 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.633736 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.634068 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.634189 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 02:23:18.634481 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.627844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:33957 (local address 127.12.244.195:47073)
0926 02:23:18.628731 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:18.628748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:18.628906 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:18.629279 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:18.629284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:18.629310 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:18.629684 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.629694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.630777 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.630784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.632252 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:18.632265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.633675 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.633682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:18.633863 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.633896 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:23:18.634008 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 02:23:18.634136 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":520,"threads_started":1}
I20260926 02:23:18.634294 28073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbf207443364e10a3a2415074046de9" candidate_uuid: "569f349128cd4c819ed58392f0de77b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a244344e47423fae3d291ad756daf5" is_pre_election: true
I20260926 02:23:18.634883 28073 raft_consensus.cc:2454] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 569f349128cd4c819ed58392f0de77b0 in term 0.
I20260926 02:23:18.635619 28141 leader_election.cc:302] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569f349128cd4c819ed58392f0de77b0, f1a244344e47423fae3d291ad756daf5; no voters:
I20260926 02:23:18.635537 27940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbf207443364e10a3a2415074046de9" candidate_uuid: "569f349128cd4c819ed58392f0de77b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" is_pre_election: true
I20260926 02:23:18.636009 28274 raft_consensus.cc:2781] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:23:18.636065 27940 raft_consensus.cc:2454] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 569f349128cd4c819ed58392f0de77b0 in term 0.
I20260926 02:23:18.636178 28274 raft_consensus.cc:482] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:23:18.636277 28274 raft_consensus.cc:3037] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:18.638639 28274 raft_consensus.cc:504] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.639377 28274 leader_election.cc:288] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [CANDIDATE]: Term 1 election: Requested vote from peers f1a244344e47423fae3d291ad756daf5 (127.12.244.194:33057), 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957)
I20260926 02:23:18.639763 28073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbf207443364e10a3a2415074046de9" candidate_uuid: "569f349128cd4c819ed58392f0de77b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a244344e47423fae3d291ad756daf5"
I20260926 02:23:18.639885 27940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbbf207443364e10a3a2415074046de9" candidate_uuid: "569f349128cd4c819ed58392f0de77b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284a97c2a29e4ac9bce9973c4cea63eb"
I20260926 02:23:18.639995 28073 raft_consensus.cc:3037] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:18.640146 27940 raft_consensus.cc:3037] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:18.642336 27940 raft_consensus.cc:2454] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569f349128cd4c819ed58392f0de77b0 in term 1.
I20260926 02:23:18.642333 28073 raft_consensus.cc:2454] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569f349128cd4c819ed58392f0de77b0 in term 1.
I20260926 02:23:18.642835 28139 leader_election.cc:302] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284a97c2a29e4ac9bce9973c4cea63eb, 569f349128cd4c819ed58392f0de77b0; no voters:
I20260926 02:23:18.643253 28274 raft_consensus.cc:2781] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:18.643697 28274 raft_consensus.cc:686] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 1 LEADER]: Becoming Leader. State: Replica: 569f349128cd4c819ed58392f0de77b0, State: Running, Role: LEADER
I20260926 02:23:18.644174 28274 consensus_queue.cc:237] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } }
I20260926 02:23:18.649108 27800 catalog_manager.cc:5495] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 569f349128cd4c819ed58392f0de77b0 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "569f349128cd4c819ed58392f0de77b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "569f349128cd4c819ed58392f0de77b0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 } health_report { overall_health: UNKNOWN } } }
I20260926 02:23:18.730890 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.725054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.12.244.195:35777)
0926 02:23:18.725280 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.725288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.725379 (+ 91us) server_negotiation.cc:407] Connection header received
0926 02:23:18.725647 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.725651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.725732 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.725901 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:23:18.726311 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.727071 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.728586 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.729007 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.729149 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.729724 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:23:18.730274 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:23:18.730445 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.730521 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.730613 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:23:18.739890 28073 raft_consensus.cc:1261] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 1 FOLLOWER]: Refusing update from remote peer 569f349128cd4c819ed58392f0de77b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:18.739935 27940 raft_consensus.cc:1261] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Refusing update from remote peer 569f349128cd4c819ed58392f0de77b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:18.740950 28276 consensus_queue.cc:1035] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a244344e47423fae3d291ad756daf5" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:18.741338 28274 consensus_queue.cc:1035] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "284a97c2a29e4ac9bce9973c4cea63eb" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 33957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:18.759023 28279 mvcc.cc:205] Tried to move back new op lower bound from 7333434977226465280 to 7333434976850362368. Current Snapshot: MvccSnapshot[applied={T|T < 7333434977226465280}]
I20260926 02:23:18.777390 27799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:18.777722 27799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:18.782284 27799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Sending DeleteTablet for 3 replicas of tablet cbbf207443364e10a3a2415074046de9
I20260926 02:23:18.783586 28185 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbf207443364e10a3a2415074046de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:18 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:44060
I20260926 02:23:18.783635 28053 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbf207443364e10a3a2415074046de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:18 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:40162
I20260926 02:23:18.785310 27920 tablet_service.cc:1514] Processing DeleteTablet for tablet cbbf207443364e10a3a2415074046de9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:18 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:34092
I20260926 02:23:18.786643 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27837
I20260926 02:23:18.798012 28288 tablet_replica.cc:330] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: stopping tablet replica
I20260926 02:23:18.800617 28288 raft_consensus.cc:2229] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:23:18.801614 28288 raft_consensus.cc:2258] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:18.809158 28290 tablet_replica.cc:330] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: stopping tablet replica
I20260926 02:23:18.810662 28288 ts_tablet_manager.cc:1905] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:18.811621 28290 raft_consensus.cc:2229] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:18.811959 28289 tablet_replica.cc:330] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: stopping tablet replica
I20260926 02:23:18.812377 28290 raft_consensus.cc:2258] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:18.812753 28289 raft_consensus.cc:2229] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:18.815539 28289 raft_consensus.cc:2258] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:18.817266 28290 ts_tablet_manager.cc:1905] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:18.817657 28289 ts_tablet_manager.cc:1905] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:18.819702 28288 ts_tablet_manager.cc:1918] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:18.819919 28288 log.cc:1196] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/cbbf207443364e10a3a2415074046de9
I20260926 02:23:18.820529 28288 ts_tablet_manager.cc:1939] T cbbf207443364e10a3a2415074046de9 P 569f349128cd4c819ed58392f0de77b0: Deleting consensus metadata
I20260926 02:23:18.822536 27783 catalog_manager.cc:4841] TS 569f349128cd4c819ed58392f0de77b0 (127.12.244.195:35777): tablet cbbf207443364e10a3a2415074046de9 (table test-table [id=1fca6fa3991543ccb4d9ca9441d2999d]) successfully deleted
I20260926 02:23:18.826076 28290 ts_tablet_manager.cc:1918] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:18.826310 28290 log.cc:1196] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/cbbf207443364e10a3a2415074046de9
I20260926 02:23:18.826987 28290 ts_tablet_manager.cc:1939] T cbbf207443364e10a3a2415074046de9 P 284a97c2a29e4ac9bce9973c4cea63eb: Deleting consensus metadata
I20260926 02:23:18.829120 27784 catalog_manager.cc:4841] TS 284a97c2a29e4ac9bce9973c4cea63eb (127.12.244.193:33957): tablet cbbf207443364e10a3a2415074046de9 (table test-table [id=1fca6fa3991543ccb4d9ca9441d2999d]) successfully deleted
I20260926 02:23:18.829530 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.787721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.12.244.193:33957)
0926 02:23:18.787956 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.787965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.794988 (+ 7023us) server_negotiation.cc:407] Connection header received
0926 02:23:18.795987 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.795996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.796080 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.796162 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:18.801936 (+ 5774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.803249 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.808446 (+ 5197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.809292 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.809400 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.817963 (+ 8563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:18.817977 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:18.817995 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:18.818257 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:18.823255 (+ 4998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.826458 (+ 3203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.826475 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.826482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.826694 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.827357 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.827362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.827364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.827853 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:18.828203 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.828843 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.829123 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:23:18.830948 27981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:18.837389 28289 ts_tablet_manager.cc:1918] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:18.837601 28289 log.cc:1196] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/cbbf207443364e10a3a2415074046de9
I20260926 02:23:18.838124 28289 ts_tablet_manager.cc:1939] T cbbf207443364e10a3a2415074046de9 P f1a244344e47423fae3d291ad756daf5: Deleting consensus metadata
I20260926 02:23:18.840096 27786 catalog_manager.cc:4841] TS f1a244344e47423fae3d291ad756daf5 (127.12.244.194:33057): tablet cbbf207443364e10a3a2415074046de9 (table test-table [id=1fca6fa3991543ccb4d9ca9441d2999d]) successfully deleted
I20260926 02:23:18.941468 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27837
I20260926 02:23:18.959259 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27990
I20260926 02:23:18.974359 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:18.960156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.12.244.194:33057)
0926 02:23:18.960343 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:18.960349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:18.960370 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:23:18.960581 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:18.960594 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:18.960659 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:18.960767 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:18.962147 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.962890 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.964418 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:18.964827 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:18.964977 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:18.967829 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:18.967845 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:18.967862 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:18.967900 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:18.971347 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.972121 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.972139 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.972141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.972278 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:18.972700 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:18.972706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:18.972711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:18.973117 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:18.973415 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:18.973768 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 02:23:18.973999 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:23:18.975024 28113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:19.067813 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27990
I20260926 02:23:19.086393 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 28121
I20260926 02:23:19.104871 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:19.087524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.12.244.195:35777)
0926 02:23:19.087751 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:19.087758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:19.087779 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:23:19.087856 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:19.087863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:19.087971 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:19.088123 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:19.089430 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:19.090678 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:19.092150 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:19.092872 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:19.093021 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:19.096271 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:19.096290 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:19.096311 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:19.096384 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:19.101000 (+ 4616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:19.102104 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:19.102115 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:19.102118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:19.102369 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:19.102839 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:19.102845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:19.102848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:19.103416 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:19.103768 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:19.104150 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 02:23:19.104404 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:23:19.105867 28245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:19.233970 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 28121
I20260926 02:23:19.253983 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 27766
I20260926 02:23:19.255404 27830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:19.379145 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 27766
I20260926 02:23:19.402354 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45941
--webserver_interface=127.12.244.254
--webserver_port=37223
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:45941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:19.776327 28296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:19.776703 28296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:19.776866 28296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:19.784247 28296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:19.784430 28296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:19.784548 28296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:19.784646 28296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:19.784734 28296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:19.794981 28296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:45941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=37223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:19.796591 28296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:19.797971 28296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:19.812836 28302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:19.813889 28305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:19.813809 28296 server_base.cc:1034] running on GCE node
W20260926 02:23:19.813838 28303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:19.815263 28296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:19.816880 28296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:19.818131 28296 hybrid_clock.cc:648] HybridClock initialized: now 1790389399818064 us; error 47 us; skew 500 ppm
I20260926 02:23:19.818517 28296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:19 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389399, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:19.825825 28296 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:23:19.827556 28296 webserver.cc:460] Webserver started at http://127.12.244.254:37223/ using document root <none> and password file <none>
I20260926 02:23:19.828272 28296 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:19.828380 28296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:19.834240 28296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:23:19.837355 28312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:19.838877 28296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:23:19.839212 28296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "ffd06c898b964614a469fddd135d9926"
format_stamp: "Formatted at 2026-09-26 02:23:13 on dist-test-slave-qfmw"
server_key: "f7c43430e9f5f6449b5e2bb3007aed38"
server_key_iv: "b7ca1ef1f760c06221f5ec5bd3cd8a6e"
server_key_version: "encryptionkey@0"
I20260926 02:23:19.840276 28296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:19.855950 28296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:19.860812 28296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:19.861146 28296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:19.881531 28296 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:45941
I20260926 02:23:19.881628 28363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:45941 every 8 connection(s)
I20260926 02:23:19.883213 28296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:23:19.888530 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 28296
I20260926 02:23:19.889387 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:33957
--local_ip_for_outbound_sockets=127.12.244.193
--tserver_master_addrs=127.12.244.254:45941
--webserver_port=41213
--webserver_interface=127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:23:19.890960 28364 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:23:19.894577 28364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Bootstrap starting.
I20260926 02:23:19.927500 28364 log.cc:824] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:19.954641 28364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:23:19.955433 28364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926: Bootstrap complete.
I20260926 02:23:19.971042 28364 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:19.972415 28364 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd06c898b964614a469fddd135d9926, State: Initialized, Role: FOLLOWER
I20260926 02:23:19.973196 28364 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:19.973476 28364 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:23:19.973618 28364 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:23:19.973805 28364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:23:19.979565 28364 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:19.980207 28364 leader_election.cc:302] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd06c898b964614a469fddd135d9926; no voters:
I20260926 02:23:19.980952 28364 leader_election.cc:288] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:23:19.981257 28368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:23:19.982273 28368 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [term 3 LEADER]: Becoming Leader. State: Replica: ffd06c898b964614a469fddd135d9926, State: Running, Role: LEADER
I20260926 02:23:19.983085 28364 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:19.982968 28368 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } }
I20260926 02:23:19.992071 28369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ffd06c898b964614a469fddd135d9926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } } }
I20260926 02:23:19.992451 28369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:19.995339 28370 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd06c898b964614a469fddd135d9926. Latest consensus state: current_term: 3 leader_uuid: "ffd06c898b964614a469fddd135d9926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd06c898b964614a469fddd135d9926" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45941 } } }
I20260926 02:23:19.995764 28370 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd06c898b964614a469fddd135d9926 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:19.999609 28377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:23:20.011225 28377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:23:20.012964 28377 catalog_manager.cc:1259] Loaded cluster ID: 16ae0ca24ced4beb9348bd4adbbc2803
I20260926 02:23:20.013094 28377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 02:23:20.015204 28377 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f31bd3299ad google::LogMessage::Fail() at ??:0
@ 0x7f31bd32c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f31bd3294de google::LogMessage::Flush() at ??:0
@ 0x7f31bd32aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f31d2d5c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f31d2d59976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f31d2de1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f31d2de1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f31bff75345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f31bff7891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f31bff78799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f31bff53eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f31c322d6db start_thread at ??:0
@ 0x7f31bc1c361f clone at ??:0
W20260926 02:23:20.337448 28366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:20.337766 28366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:20.337930 28366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:20.345479 28366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:20.345614 28366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:20.345835 28366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:23:20.356355 28366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:33957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=41213
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:20.357839 28366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:20.359258 28366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:20.373340 28408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:20.373649 28410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:20.373579 28407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:20.374677 28366 server_base.cc:1034] running on GCE node
I20260926 02:23:20.375430 28366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:20.376554 28366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:20.377807 28366 hybrid_clock.cc:648] HybridClock initialized: now 1790389400377761 us; error 72 us; skew 500 ppm
I20260926 02:23:20.378142 28366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:20 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389400, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:20.381536 28366 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:23:20.383327 28366 webserver.cc:460] Webserver started at http://127.12.244.193:41213/ using document root <none> and password file <none>
I20260926 02:23:20.384096 28366 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:20.384250 28366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:20.390031 28366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 02:23:20.393446 28417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:20.395188 28366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:23:20.395408 28366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "284a97c2a29e4ac9bce9973c4cea63eb"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "670689d5cda547516c3b36db7fa11143"
server_key_iv: "9a81ec73a028ed240c0d9a245ad70e8d"
server_key_version: "encryptionkey@0"
I20260926 02:23:20.396516 28366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:20.416662 28366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:20.421365 28366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:20.421686 28366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:20.422811 28366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:20.424392 28366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:20.424525 28366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:20.424679 28366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:20.424774 28366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:20.460925 28366 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:33957
I20260926 02:23:20.460986 28529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:33957 every 8 connection(s)
I20260926 02:23:20.462740 28366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:23:20.467945 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:20.465999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:58369)
0926 02:23:20.466763 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:20.466986 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":275,"threads_started":1}
W20260926 02:23:20.468679 28530 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:45941 (0 consecutive failures): Network error: Failed to ping master at 127.12.244.254:45941: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
I20260926 02:23:20.469720 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 28366
I20260926 02:23:20.469756 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:20.469256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:55245)
0926 02:23:20.469443 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:20.469545 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:23:20.470512 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:33057
--local_ip_for_outbound_sockets=127.12.244.194
--tserver_master_addrs=127.12.244.254:45941
--webserver_port=40483
--webserver_interface=127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:23:20.470830 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:20.470382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:37693)
0926 02:23:20.470550 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:20.470667 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:23:20.471179 28530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:23:20.819622 28533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:20.819944 28533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:20.820149 28533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:20.827836 28533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:20.827967 28533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:20.828181 28533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:23:20.838322 28533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:33057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=40483
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:20.839790 28533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:20.841179 28533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:20.855626 28538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:20.855624 28539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:20.855855 28541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:20.856629 28533 server_base.cc:1034] running on GCE node
I20260926 02:23:20.857316 28533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:20.858351 28533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:20.859617 28533 hybrid_clock.cc:648] HybridClock initialized: now 1790389400859554 us; error 33 us; skew 500 ppm
I20260926 02:23:20.859961 28533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:20 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389400, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:20.863587 28533 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:23:20.865159 28533 webserver.cc:460] Webserver started at http://127.12.244.194:40483/ using document root <none> and password file <none>
I20260926 02:23:20.865845 28533 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:20.865947 28533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:20.871608 28533 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:23:20.874627 28549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:20.876215 28533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:23:20.876379 28533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "f1a244344e47423fae3d291ad756daf5"
format_stamp: "Formatted at 2026-09-26 02:23:14 on dist-test-slave-qfmw"
server_key: "2a6acfb2f36aab7aa036103845542c18"
server_key_iv: "7713bd670168b256983383ac7232e49e"
server_key_version: "encryptionkey@0"
I20260926 02:23:20.877379 28533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:20.896826 28533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:20.901796 28533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:20.902043 28533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:20.903070 28533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:20.904630 28533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:20.904732 28533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:20.904814 28533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:20.904870 28533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:20.941232 28533 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:33057
I20260926 02:23:20.941315 28662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:33057 every 8 connection(s)
I20260926 02:23:20.942889 28533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:23:20.944416 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 28533
I20260926 02:23:20.945173 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:35777
--local_ip_for_outbound_sockets=127.12.244.195
--tserver_master_addrs=127.12.244.254:45941
--webserver_port=40993
--webserver_interface=127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:45417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:23:20.953254 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:20.946393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:47435)
0926 02:23:20.951720 (+ 5327us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:20.952076 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4887,"thread_start_us":4639,"threads_started":1}
W20260926 02:23:20.954128 28663 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:45941 (0 consecutive failures): Network error: Failed to ping master at 127.12.244.254:45941: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
I20260926 02:23:20.955303 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:20.954782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:57581)
0926 02:23:20.954999 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:20.955135 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 02:23:20.956390 28665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:20.956048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:41129)
0926 02:23:20.956189 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:20.956271 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:23:20.956743 28663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:23:21.293617 28666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:21.293982 28666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:21.294162 28666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:21.301878 28666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:21.302057 28666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:21.302235 28666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:23:21.312440 28666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:45417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:35777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=40993
--tserver_master_addrs=127.12.244.254:45941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:21.314057 28666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:21.316093 28666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:21.330431 28674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:21.330447 28672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:21.330582 28671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:21.331507 28666 server_base.cc:1034] running on GCE node
I20260926 02:23:21.332247 28666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:21.333303 28666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:21.334566 28666 hybrid_clock.cc:648] HybridClock initialized: now 1790389401334481 us; error 77 us; skew 500 ppm
I20260926 02:23:21.334944 28666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:21 dist-test-slave-qfmw krb5kdc[27208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389401, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:21.338248 28666 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:23:21.339846 28666 webserver.cc:460] Webserver started at http://127.12.244.195:40993/ using document root <none> and password file <none>
I20260926 02:23:21.340584 28666 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:21.340735 28666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:21.346485 28666 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:23:21.349494 28681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:21.350929 28666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:23:21.351217 28666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "569f349128cd4c819ed58392f0de77b0"
format_stamp: "Formatted at 2026-09-26 02:23:15 on dist-test-slave-qfmw"
server_key: "4e490c2636219de62379f8d8e954ef33"
server_key_iv: "426b553d20bf23d0b67619a6637bc881"
server_key_version: "encryptionkey@0"
I20260926 02:23:21.352298 28666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:21.370760 28666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:21.375371 28666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:21.375618 28666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:21.376611 28666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:21.377933 28666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:21.378023 28666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:21.378116 28666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:21.378170 28666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:21.412668 28666 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:35777
I20260926 02:23:21.412668 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:35777 every 8 connection(s)
I20260926 02:23:21.414314 28666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:23:21.419787 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:21.417699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:33527)
0926 02:23:21.418469 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:21.418753 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":311,"threads_started":1}
W20260926 02:23:21.420513 28795 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:45941 (0 consecutive failures): Network error: Failed to ping master at 127.12.244.254:45941: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
I20260926 02:23:21.421451 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:21.421098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:37913)
0926 02:23:21.421267 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:21.421345 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 02:23:21.422371 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:21.422015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:43339)
0926 02:23:21.422173 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:21.422257 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:23:21.422406 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 28666
W20260926 02:23:21.422730 28795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:23:21.472656 28799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:21.471750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:57273)
0926 02:23:21.472246 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:21.472389 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":241,"threads_started":1}
I20260926 02:23:21.958335 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:21.957353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:60557)
0926 02:23:21.957943 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:21.958096 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":266,"threads_started":1}
I20260926 02:23:22.424476 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:22.423471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:37771)
0926 02:23:22.424007 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:22.424155 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
I20260926 02:23:22.474412 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:22.473670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:36471)
0926 02:23:22.474147 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:22.474240 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":228,"threads_started":1}
I20260926 02:23:22.960493 28803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:22.959569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:55609)
0926 02:23:22.960169 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:22.960297 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":297,"threads_started":1}
I20260926 02:23:23.426417 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:23.425554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:37865)
0926 02:23:23.426168 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:23.426250 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":309,"threads_started":1}
I20260926 02:23:23.476130 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:23.475378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:53289)
0926 02:23:23.475843 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:23.475931 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":248,"threads_started":1}
I20260926 02:23:23.962303 28807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:23.961575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:44867)
0926 02:23:23.962026 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:23.962125 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260926 02:23:24.428098 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:24.427352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:41783)
0926 02:23:24.427830 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:24.427919 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":253,"threads_started":1}
I20260926 02:23:24.477919 28809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:24.477128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:41851)
0926 02:23:24.477639 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:24.477734 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260926 02:23:24.964355 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:24.963567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:51493)
0926 02:23:24.964086 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:24.964178 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260926 02:23:25.429684 28811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:25.428992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:54779)
0926 02:23:25.429412 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:25.429491 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":216,"threads_started":1}
I20260926 02:23:25.479853 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:25.478996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:37427)
0926 02:23:25.479553 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:25.479679 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":295,"threads_started":1}
I20260926 02:23:25.966256 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:25.965419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:37005)
0926 02:23:25.965972 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:25.966057 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":300,"threads_started":1}
I20260926 02:23:26.431684 28814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:26.430780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:54149)
0926 02:23:26.431385 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:26.431484 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":330,"threads_started":1}
I20260926 02:23:26.481631 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:26.480827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:52159)
0926 02:23:26.481354 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:26.481455 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":322,"threads_started":1}
I20260926 02:23:26.968070 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:26.967334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:33813)
0926 02:23:26.967799 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:26.967897 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260926 02:23:27.433449 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:27.432687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:44007)
0926 02:23:27.433186 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:27.433285 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":259,"threads_started":1}
I20260926 02:23:27.483556 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:27.482658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:57161)
0926 02:23:27.483255 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:27.483348 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":227,"threads_started":1}
I20260926 02:23:27.970146 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:27.969235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:57771)
0926 02:23:27.969803 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:27.969893 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":297,"threads_started":1}
I20260926 02:23:28.435483 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:28.434576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:40309)
0926 02:23:28.435074 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:28.435248 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":215,"threads_started":1}
I20260926 02:23:28.485435 28821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:28.484546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:42147)
0926 02:23:28.485138 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:28.485230 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":245,"threads_started":1}
I20260926 02:23:28.972208 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:28.971296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:52215)
0926 02:23:28.971892 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:28.971999 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":288,"threads_started":1}
I20260926 02:23:29.437395 28823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:29.436567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:34935)
0926 02:23:29.437135 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:29.437231 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":276,"threads_started":1}
I20260926 02:23:29.487164 28824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:29.486332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:58961)
0926 02:23:29.486832 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:29.486921 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":247,"threads_started":1}
I20260926 02:23:29.974129 28825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:29.973328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:45145)
0926 02:23:29.973854 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:29.973945 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":232,"threads_started":1}
I20260926 02:23:30.443867 28826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:30.438409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:50771)
0926 02:23:30.443583 (+ 5174us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:30.443679 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5041,"thread_start_us":4858,"threads_started":1}
I20260926 02:23:30.489149 28827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:30.488187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:38987)
0926 02:23:30.488855 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:30.488959 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":416,"threads_started":1}
I20260926 02:23:30.976150 28828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:30.975364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:54627)
0926 02:23:30.975840 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:30.975971 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260926 02:23:31.445716 28829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:31.444929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:35707)
0926 02:23:31.445465 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:31.445546 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":318,"threads_started":1}
I20260926 02:23:31.491189 28830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:31.490220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:34969)
0926 02:23:31.490826 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:31.490951 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":379,"threads_started":1}
I20260926 02:23:31.978025 28831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:31.977169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:54169)
0926 02:23:31.977724 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:31.977825 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":316,"threads_started":1}
I20260926 02:23:32.447474 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:32.446695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:59091)
0926 02:23:32.447182 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:32.447280 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260926 02:23:32.492903 28833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:32.492232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:49419)
0926 02:23:32.492661 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:32.492747 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260926 02:23:32.980011 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:32.979269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:40767)
0926 02:23:32.979766 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:32.979852 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260926 02:23:33.449421 28835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:33.448636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:60351)
0926 02:23:33.449167 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:33.449259 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":258,"threads_started":1}
I20260926 02:23:33.494773 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:33.493964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:46851)
0926 02:23:33.494446 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:33.494552 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":228,"threads_started":1}
I20260926 02:23:33.982038 28837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:33.981225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:34097)
0926 02:23:33.981740 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:33.981845 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":234,"threads_started":1}
I20260926 02:23:34.451332 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:34.450441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:48267)
0926 02:23:34.450974 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:34.451074 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
I20260926 02:23:34.496508 28839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:34.495775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:55201)
0926 02:23:34.496239 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:34.496326 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260926 02:23:34.983840 28840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:34.983005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:59711)
0926 02:23:34.983542 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:34.983654 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":306,"threads_started":1}
I20260926 02:23:35.453081 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:35.452288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:51551)
0926 02:23:35.452753 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:35.452885 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":220,"threads_started":1}
I20260926 02:23:35.498579 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:35.497802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.193:36347)
0926 02:23:35.498291 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:35.498396 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
I20260926 02:23:35.985774 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:35.985030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.194:60515)
0926 02:23:35.985477 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:35.985589 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":232,"threads_started":1}
I20260926 02:23:36.430253 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 28366
I20260926 02:23:36.455461 28845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:36.454546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45941 (local address 127.12.244.195:38269)
0926 02:23:36.455142 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:36.455262 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":312,"threads_started":1}
I20260926 02:23:36.459458 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:36.432099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.12.244.193:33957)
0926 02:23:36.432733 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:36.432756 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:36.432810 (+ 54us) server_negotiation.cc:407] Connection header received
0926 02:23:36.435434 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:36.435452 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:36.436155 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:36.436582 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:36.438255 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:36.439644 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:36.441031 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:36.441776 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:36.441996 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:36.451237 (+ 9241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:36.451249 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:36.451270 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:36.451330 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:36.455265 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:36.456353 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:36.456363 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:36.456372 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:36.456596 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:36.457099 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:36.457104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:36.457114 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:36.457742 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:36.458105 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:36.458611 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 02:23:36.459150 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":291,"threads_started":1}
I20260926 02:23:36.460932 28525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:36.548758 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 28366
I20260926 02:23:36.566727 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 28533
I20260926 02:23:36.586063 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:36.567731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.12.244.194:33057)
0926 02:23:36.568294 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:36.568309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:36.568349 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:23:36.569982 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:36.569993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:36.570449 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:36.570764 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:36.572338 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:36.573678 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:36.575197 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:36.575970 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:36.576122 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:36.578898 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:36.578907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:36.578932 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:36.578984 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:36.582595 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:36.583557 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:36.583566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:36.583576 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:36.583774 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:36.584170 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:36.584173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:36.584175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:36.584769 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:36.585025 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:36.585406 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 02:23:36.585765 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":249,"threads_started":1}
I20260926 02:23:36.587625 28658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:36.673985 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 28533
I20260926 02:23:36.692612 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 28666
I20260926 02:23:36.715674 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:36.693651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.12.244.195:35777)
0926 02:23:36.694339 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:36.694352 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:36.694398 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:23:36.696196 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:36.696206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:36.696705 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:36.697043 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:36.699440 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:36.701525 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:36.703651 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:36.705021 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:36.705282 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:36.708330 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:36.708339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:36.708357 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:36.708407 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:36.711880 (+ 3473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:36.712830 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:36.712838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:36.712847 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:36.713050 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:36.713428 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:36.713432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:36.713434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:36.714031 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:36.714296 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:36.714813 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 02:23:36.715350 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":356,"threads_started":1}
I20260926 02:23:36.717338 28790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:36.802086 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 28666
2026-09-26T02:23:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25052 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:23:36 dist-test-slave-qfmw krb5kdc[28853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:23:36 dist-test-slave-qfmw krb5kdc[28853](info): set up 2 sockets
Sep 26 02:23:36 dist-test-slave-qfmw krb5kdc[28853](info): commencing operation
krb5kdc: starting...
W20260926 02:23:37.918205 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:23:38 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:23:38.179282 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36095
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:38.539829 28866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:38.540175 28866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:38.540266 28866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:38.547869 28866 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:23:38.547995 28866 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:23:38.548035 28866 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:23:38.548080 28866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:23:38.548138 28866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:38.548211 28866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:38.548259 28866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:23:38.548305 28866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:23:38.558718 28866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:36095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:38.560181 28866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:38.561558 28866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:38.575402 28872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:38.575533 28874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:38.575553 28871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:38.575891 28866 server_base.cc:1034] running on GCE node
I20260926 02:23:38.577021 28866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:38.578555 28866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:38.579818 28866 hybrid_clock.cc:648] HybridClock initialized: now 1790389418579750 us; error 38 us; skew 500 ppm
I20260926 02:23:38.580155 28866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:38 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389418, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:38.583595 28866 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:23:38.585294 28866 webserver.cc:460] Webserver started at http://127.12.244.254:40615/ using document root <none> and password file <none>
I20260926 02:23:38.586104 28866 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:38.586248 28866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:38.586601 28866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:38.589862 28866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "f82c312c1cd34a58998741ec779e2590"
format_stamp: "Formatted at 2026-09-26 02:23:38 on dist-test-slave-qfmw"
server_key: "20eaf7be57909b073eba7b56c54fa43e"
server_key_iv: "b8a35f95ce62f9aa2cb71fcd165ecbb8"
server_key_version: "encryptionkey@0"
I20260926 02:23:38.590721 28866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "f82c312c1cd34a58998741ec779e2590"
format_stamp: "Formatted at 2026-09-26 02:23:38 on dist-test-slave-qfmw"
server_key: "20eaf7be57909b073eba7b56c54fa43e"
server_key_iv: "b8a35f95ce62f9aa2cb71fcd165ecbb8"
server_key_version: "encryptionkey@0"
I20260926 02:23:38.596724 28866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 02:23:38.600948 28881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:38.602620 28866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:23:38.602850 28866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "f82c312c1cd34a58998741ec779e2590"
format_stamp: "Formatted at 2026-09-26 02:23:38 on dist-test-slave-qfmw"
server_key: "20eaf7be57909b073eba7b56c54fa43e"
server_key_iv: "b8a35f95ce62f9aa2cb71fcd165ecbb8"
server_key_version: "encryptionkey@0"
I20260926 02:23:38.603137 28866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:38.621081 28866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:38.626447 28866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:38.626807 28866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:38.647464 28866 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:36095
I20260926 02:23:38.647512 28933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:36095 every 8 connection(s)
I20260926 02:23:38.649003 28866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:23:38.653199 28934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:38.655547 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 28866
I20260926 02:23:38.655805 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:23:38.656649 13267 external_mini_cluster.cc:1428] Setting key 0ac0dd947dbab12d1490517cef658e14
I20260926 02:23:38.667732 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Bootstrap starting.
I20260926 02:23:38.672036 28934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Neither blocks nor log segments found. Creating new log.
Sep 26 02:23:38 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:38.673302 28934 log.cc:824] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:38.676959 28934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: No bootstrap required, opened a new log
I20260926 02:23:38.682533 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:38.658967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.12.244.254:36095)
0926 02:23:38.660041 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:38.660053 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:38.660107 (+ 54us) server_negotiation.cc:407] Connection header received
0926 02:23:38.661915 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:38.661941 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:38.662396 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:38.662782 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:38.664812 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:38.666365 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:38.668395 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:38.669443 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:38.669671 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:38.674330 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:38.674344 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:38.674362 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:38.674405 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:38.677746 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:38.678610 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:38.678618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:38.678627 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:38.678832 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:38.679360 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:38.679364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:38.679366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:38.680016 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:38.680363 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:38.680859 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 02:23:38.681362 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":413,"threads_started":1}
I20260926 02:23:38.686128 28934 raft_consensus.cc:348] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:23:38.686614 28934 raft_consensus.cc:374] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:38.686731 28934 raft_consensus.cc:729] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82c312c1cd34a58998741ec779e2590, State: Initialized, Role: FOLLOWER
I20260926 02:23:38.687404 28934 consensus_queue.cc:260] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:23:38.687685 28934 raft_consensus.cc:388] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:23:38.687834 28934 raft_consensus.cc:482] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:23:38.688012 28934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:38.690304 28934 raft_consensus.cc:504] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:23:38.690866 28934 leader_election.cc:302] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f82c312c1cd34a58998741ec779e2590; no voters:
I20260926 02:23:38.691713 28934 leader_election.cc:288] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:23:38.691813 28939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:38.692648 28939 raft_consensus.cc:686] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 LEADER]: Becoming Leader. State: Replica: f82c312c1cd34a58998741ec779e2590, State: Running, Role: LEADER
I20260926 02:23:38.693331 28939 consensus_queue.cc:237] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:23:38.694286 28934 sys_catalog.cc:564] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:23:38.697046 28941 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f82c312c1cd34a58998741ec779e2590. Latest consensus state: current_term: 1 leader_uuid: "f82c312c1cd34a58998741ec779e2590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } } }
I20260926 02:23:38.698117 28941 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: This master's current role is: LEADER
I20260926 02:23:38.703859 28940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f82c312c1cd34a58998741ec779e2590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } } }
I20260926 02:23:38.704263 28940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: This master's current role is: LEADER
W20260926 02:23:38.711678 28954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:23:38.711825 28954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:23:38.711990 28945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:23:38.714447 28945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:23:38.725719 28945 catalog_manager.cc:1347] Generated new cluster ID: e122588663b64384955ebdbf21444e41
I20260926 02:23:38.725881 28945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:23:38.738686 28945 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:23:38.740226 28945 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:23:38.751332 28945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Generated new TSK 0
I20260926 02:23:38.752359 28945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:23:38.815937 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:36095
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:39.184844 28960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:39.185153 28960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:39.185317 28960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:39.192996 28960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:39.193131 28960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:39.193332 28960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:23:39.204232 28960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:39.205693 28960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:39.207307 28960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:39.221823 28965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:39.221732 28966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:39.221974 28968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:39.223384 28960 server_base.cc:1034] running on GCE node
I20260926 02:23:39.224094 28960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:39.225169 28960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:39.226388 28960 hybrid_clock.cc:648] HybridClock initialized: now 1790389419226316 us; error 64 us; skew 500 ppm
I20260926 02:23:39.226794 28960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:39 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389419, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:39.230473 28960 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:23:39.232273 28960 webserver.cc:460] Webserver started at http://127.12.244.193:43173/ using document root <none> and password file <none>
I20260926 02:23:39.233052 28960 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:39.233192 28960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:39.233539 28960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:39.236847 28960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "d56b6a51b92842e4b749c1d74ac1d4e3"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "749b2c8ef0fb8e3c8af878a8f3e051e0"
server_key_iv: "4217be066fefbdeac08716c6c1b0814a"
server_key_version: "encryptionkey@0"
I20260926 02:23:39.237690 28960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "d56b6a51b92842e4b749c1d74ac1d4e3"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "749b2c8ef0fb8e3c8af878a8f3e051e0"
server_key_iv: "4217be066fefbdeac08716c6c1b0814a"
server_key_version: "encryptionkey@0"
I20260926 02:23:39.243580 28960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:23:39.247572 28975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:39.249130 28960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:23:39.249325 28960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "d56b6a51b92842e4b749c1d74ac1d4e3"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "749b2c8ef0fb8e3c8af878a8f3e051e0"
server_key_iv: "4217be066fefbdeac08716c6c1b0814a"
server_key_version: "encryptionkey@0"
I20260926 02:23:39.249588 28960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:39.276295 28960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:39.281213 28960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:39.281544 28960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:39.282625 28960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:39.284240 28960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:39.284320 28960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:39.284477 28960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:39.284538 28960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:39.321913 28960 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:38281
I20260926 02:23:39.321977 29088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:38281 every 8 connection(s)
I20260926 02:23:39.323447 28960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:23:39.325028 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 28960
I20260926 02:23:39.325258 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:23:39.326062 13267 external_mini_cluster.cc:1428] Setting key 5eb106a4dad1a416a0d25282d9ca7bca
Sep 26 02:23:39 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389419, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:39.347966 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:39.326559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:58717 (local address 127.12.244.254:36095)
0926 02:23:39.327160 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:39.327166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:39.328502 (+ 1336us) server_negotiation.cc:407] Connection header received
0926 02:23:39.330683 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:39.330688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:39.330785 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:39.330929 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:39.332826 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.333770 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.336164 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.336646 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.336761 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:39.341477 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:39.341483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:39.341486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:39.341527 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:39.344499 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:39.345420 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:39.345425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:39.345428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:39.345561 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:39.346053 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:39.346057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:39.346059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:39.346336 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:39.346502 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:39.347333 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 02:23:39.347590 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":262,"threads_started":1}
I20260926 02:23:39.349031 29092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:39.327374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:58717)
0926 02:23:39.328319 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:39.328377 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:39.330339 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:39.331267 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:39.331277 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:39.331858 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:39.332597 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:39.332628 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.334006 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.334011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:39.335907 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:39.335932 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.336927 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.336936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:39.337308 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:39.338264 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:39.338302 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:39.341170 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:39.344698 (+ 3528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:39.344709 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:39.344733 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:39.345256 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:39.345693 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:39.345698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:39.345702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:39.345914 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:39.346480 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:39.346489 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:39.346879 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 02:23:39.347219 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 02:23:39.347842 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":389,"threads_started":1}
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
I20260926 02:23:39.350901 29089 heartbeater.cc:347] Connected to a master server at 127.12.244.254:36095
I20260926 02:23:39.351264 29089 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:39.352027 29089 heartbeater.cc:511] Master 127.12.244.254:36095 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
I20260926 02:23:39.354665 28899 ts_manager.cc:194] Registered new tserver with Master: d56b6a51b92842e4b749c1d74ac1d4e3 (127.12.244.193:38281)
I20260926 02:23:39.356349 28899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:58717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:23:39.383081 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:36095
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:39.738425 29095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:39.738792 29095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:39.738976 29095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:39.746649 29095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:39.746838 29095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:39.747038 29095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:23:39.757421 29095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:39.758853 29095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:39.760356 29095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:39.774540 29100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:39.774508 29103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:39.774655 29101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:39.776401 29095 server_base.cc:1034] running on GCE node
I20260926 02:23:39.777173 29095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:39.778304 29095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:39.779554 29095 hybrid_clock.cc:648] HybridClock initialized: now 1790389419779512 us; error 36 us; skew 500 ppm
I20260926 02:23:39.779934 29095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:39 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389419, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:39.783645 29095 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:23:39.785351 29095 webserver.cc:460] Webserver started at http://127.12.244.194:34253/ using document root <none> and password file <none>
I20260926 02:23:39.786136 29095 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:39.786293 29095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:39.786638 29095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:39.789855 29095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "ecc901c868fb4be8a7a2eeeab3b72059"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "43ce1f0d57a4962bfb5f314bfa37b45b"
server_key_iv: "5a459a58191cadf686f64b1d6bde9af4"
server_key_version: "encryptionkey@0"
I20260926 02:23:39.790704 29095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "ecc901c868fb4be8a7a2eeeab3b72059"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "43ce1f0d57a4962bfb5f314bfa37b45b"
server_key_iv: "5a459a58191cadf686f64b1d6bde9af4"
server_key_version: "encryptionkey@0"
I20260926 02:23:39.796634 29095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 02:23:39.800645 29110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:39.802151 29095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:23:39.802366 29095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "ecc901c868fb4be8a7a2eeeab3b72059"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "43ce1f0d57a4962bfb5f314bfa37b45b"
server_key_iv: "5a459a58191cadf686f64b1d6bde9af4"
server_key_version: "encryptionkey@0"
I20260926 02:23:39.802685 29095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:39.815758 29095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:39.821334 29095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:39.821695 29095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:39.822762 29095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:39.824370 29095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:39.824501 29095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:39.824664 29095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:39.824750 29095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:39.862676 29095 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:40775
I20260926 02:23:39.862773 29223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:40775 every 8 connection(s)
I20260926 02:23:39.864188 29095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:23:39.870282 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 29095
I20260926 02:23:39.870486 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:23:39.871120 13267 external_mini_cluster.cc:1428] Setting key 69e435277d8ebc01d1751b61d01d9e71
Sep 26 02:23:39 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389419, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:39.890532 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:39.867340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:40541 (local address 127.12.244.254:36095)
0926 02:23:39.867872 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:39.867877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:39.868971 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 02:23:39.871754 (+ 2783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:39.871759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:39.871894 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:39.872094 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:39.874437 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.875284 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.878142 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.878615 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.878732 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:39.883848 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:39.883854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:39.883857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:39.883899 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:39.886641 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:39.887799 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:39.887804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:39.887806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:39.887952 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:39.888539 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:39.888543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:39.888545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:39.888856 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:39.888988 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:39.890048 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0926 02:23:39.890264 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":298,"threads_started":1}
I20260926 02:23:39.892267 29227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:39.867909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:40541)
0926 02:23:39.868762 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:39.868811 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:39.871315 (+ 2504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:39.872604 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:39.872618 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:39.873298 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:39.874204 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:39.874239 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.875463 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.875472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:39.877882 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:39.877907 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:39.878902 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:39.878912 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:39.879319 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:39.880245 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:39.880283 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:39.883556 (+ 3273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:39.886833 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:39.886847 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:39.886871 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:39.887636 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:39.888101 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:39.888108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:39.888115 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:39.888390 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:39.888999 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:39.889011 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:39.889565 (+ 554us) client_negotiation.cc:769] Sending connection context
0926 02:23:39.889904 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 02:23:39.890548 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":339,"threads_started":1}
I20260926 02:23:39.894187 29224 heartbeater.cc:347] Connected to a master server at 127.12.244.254:36095
I20260926 02:23:39.894584 29224 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:39.895641 29224 heartbeater.cc:511] Master 127.12.244.254:36095 requested a full tablet report, sending...
I20260926 02:23:39.897310 28899 ts_manager.cc:194] Registered new tserver with Master: ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775)
I20260926 02:23:39.898195 28899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:40541
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:23:39.931053 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:36095
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:23:40.298591 29230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:23:40.298966 29230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:23:40.299171 29230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:23:40.307504 29230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:23:40.307687 29230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:23:40.307916 29230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:23:40.318338 29230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:23:40.319785 29230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:23:40.321061 29230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:23:40.335716 29235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:40.335819 29236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:23:40.335963 29238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:23:40.337535 29230 server_base.cc:1034] running on GCE node
I20260926 02:23:40.338352 29230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:23:40.339531 29230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:23:40.340782 29230 hybrid_clock.cc:648] HybridClock initialized: now 1790389420340727 us; error 57 us; skew 500 ppm
I20260926 02:23:40.341156 29230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:23:40 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389420, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:23:40.344674 29230 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:23:40.346271 29230 webserver.cc:460] Webserver started at http://127.12.244.195:39281/ using document root <none> and password file <none>
I20260926 02:23:40.347012 29230 fs_manager.cc:359] Metadata directory not provided
I20260926 02:23:40.347213 29230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:23:40.347543 29230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:23:40.350654 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "f995a6fd37714b8f8ab427d21fa28731"
format_stamp: "Formatted at 2026-09-26 02:23:40 on dist-test-slave-qfmw"
server_key: "e83137734c93ecb4d0e5438ba63840e6"
server_key_iv: "a1413a3a17036a62fb422640b89bb879"
server_key_version: "encryptionkey@0"
I20260926 02:23:40.351599 29230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "f995a6fd37714b8f8ab427d21fa28731"
format_stamp: "Formatted at 2026-09-26 02:23:40 on dist-test-slave-qfmw"
server_key: "e83137734c93ecb4d0e5438ba63840e6"
server_key_iv: "a1413a3a17036a62fb422640b89bb879"
server_key_version: "encryptionkey@0"
I20260926 02:23:40.357506 29230 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260926 02:23:40.360598 29089 heartbeater.cc:503] Master 127.12.244.254:36095 was elected leader, sending a full tablet report...
I20260926 02:23:40.361642 29245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:40.363487 29230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:23:40.363675 29230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "f995a6fd37714b8f8ab427d21fa28731"
format_stamp: "Formatted at 2026-09-26 02:23:40 on dist-test-slave-qfmw"
server_key: "e83137734c93ecb4d0e5438ba63840e6"
server_key_iv: "a1413a3a17036a62fb422640b89bb879"
server_key_version: "encryptionkey@0"
I20260926 02:23:40.363885 29230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:23:40.377723 29230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:23:40.382189 29230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:23:40.382500 29230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:23:40.383703 29230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:23:40.385174 29230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:23:40.385247 29230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:40.385392 29230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:23:40.385444 29230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:23:40.423147 29230 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:38483
I20260926 02:23:40.423301 29358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:38483 every 8 connection(s)
I20260926 02:23:40.424583 29230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:23:40.426683 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 29230
I20260926 02:23:40.426894 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:23:40.427517 13267 external_mini_cluster.cc:1428] Setting key c21b1d5966b9c69efacf69a18c126acc
Sep 26 02:23:40 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389420, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:23:40.447749 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.427678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:36197 (local address 127.12.244.254:36095)
0926 02:23:40.428262 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.428268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.429342 (+ 1074us) server_negotiation.cc:407] Connection header received
0926 02:23:40.431890 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.431895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.432000 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.432149 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:40.433864 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.434673 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.436391 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.436822 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.436978 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.440740 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:40.440746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:40.440749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:40.440787 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:40.444357 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.445251 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.445255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.445258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.445390 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.445910 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.445915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.445917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.446204 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:40.446329 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.447260 (+ 931us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.447461 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":301,"threads_started":1}
I20260926 02:23:40.448941 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.428327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:36197)
0926 02:23:40.429184 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:40.429228 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:40.431584 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:40.432424 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:40.432433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:40.433008 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:23:40.433663 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.433681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.434837 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.434847 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.436237 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.436250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.437085 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.437092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.437344 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.438070 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:23:40.438094 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:23:40.440480 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:23:40.444534 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:40.444544 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:40.444561 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:40.445099 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:40.445533 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:23:40.445537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:23:40.445539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:23:40.445756 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:23:40.446348 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:23:40.446355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:23:40.446789 (+ 434us) client_negotiation.cc:769] Sending connection context
0926 02:23:40.447049 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 02:23:40.447581 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
I20260926 02:23:40.450501 29359 heartbeater.cc:347] Connected to a master server at 127.12.244.254:36095
I20260926 02:23:40.450855 29359 heartbeater.cc:464] Registering TS with master...
I20260926 02:23:40.451805 29359 heartbeater.cc:511] Master 127.12.244.254:36095 requested a full tablet report, sending...
I20260926 02:23:40.453590 28899 ts_manager.cc:194] Registered new tserver with Master: f995a6fd37714b8f8ab427d21fa28731 (127.12.244.195:38483)
I20260926 02:23:40.454476 28899 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:36197
I20260926 02:23:40.457037 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:23:40.475925 29361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.461911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.12.244.254:36095)
0926 02:23:40.462165 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.462175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.462552 (+ 377us) server_negotiation.cc:407] Connection header received
0926 02:23:40.462688 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.462691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.462774 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.462863 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:40.464435 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.465196 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.466671 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.467081 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.467265 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.470040 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:40.470045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:40.470049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:40.470097 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:40.472861 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.473778 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.473782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.473785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.473921 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.474293 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.474296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.474298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.474657 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:40.474850 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.475312 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.475556 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:23:40.480501 28899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:23:40.487653 28899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:23:40.514367 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.504320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.12.244.194:40775)
0926 02:23:40.505272 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.505284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.505319 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:23:40.505440 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.505448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.505713 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.505921 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:40.506776 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.508458 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.511077 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.512617 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.512798 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.513026 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.513891 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.514076 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":610,"threads_started":1}
I20260926 02:23:40.514452 29373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.501036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:38483 (local address 127.0.0.1:48378)
0926 02:23:40.502550 (+ 1514us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:40.502593 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:40.502772 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:40.506356 (+ 3584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:40.506370 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:40.506402 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:40.507070 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.507082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.509111 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.509119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.511233 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.511253 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.513253 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.513258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.513504 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.513580 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 02:23:40.513737 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:23:40.513965 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"mutex_wait_us":62,"thread_start_us":1023,"threads_started":1}
I20260926 02:23:40.514891 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.501100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:38281 (local address 127.0.0.1:44268)
0926 02:23:40.505255 (+ 4155us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:40.505278 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:40.505409 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:40.506222 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:40.506226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:40.506259 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:40.506845 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.506865 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.510033 (+ 3168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.510042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.511951 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.511962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.513800 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.513807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.513967 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.514001 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:23:40.514095 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 02:23:40.514234 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2735,"mutex_wait_us":7,"thread_start_us":2595,"threads_started":1}
I20260926 02:23:40.515170 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.501825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.12.244.193:38281)
0926 02:23:40.503513 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.503525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.505543 (+ 2018us) server_negotiation.cc:407] Connection header received
0926 02:23:40.505654 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.505661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.505942 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.506126 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:40.507058 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.509877 (+ 2819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.512163 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.513574 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.513796 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.513979 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.514153 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.514305 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1426,"thread_start_us":1282,"threads_started":1}
I20260926 02:23:40.515352 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.503753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:40775 (local address 127.0.0.1:58344)
0926 02:23:40.504456 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:40.504478 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:40.504645 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:40.505935 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:40.505943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:40.505973 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:40.506479 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.506493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.508611 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.508617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.510913 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.510925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.513180 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.513189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.513467 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.513580 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 02:23:40.513737 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:23:40.514760 (+ 1023us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":857,"thread_start_us":733,"threads_started":1}
I20260926 02:23:40.515590 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.501216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.12.244.195:38483)
0926 02:23:40.505731 (+ 4515us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.505739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.505766 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:23:40.505869 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.505876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.506070 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.506622 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:40.507276 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.508908 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.511405 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.512791 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.512973 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.513128 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.515016 (+ 1888us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.515225 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4323,"thread_start_us":4151,"threads_started":1}
I20260926 02:23:40.519315 29293 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0ff3bf39984d4c94665ecd3595b412 (DEFAULT_TABLE table=test-table [id=ff35316d9ea54635b3d65d1a57a4d298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:40.519327 29023 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0ff3bf39984d4c94665ecd3595b412 (DEFAULT_TABLE table=test-table [id=ff35316d9ea54635b3d65d1a57a4d298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:40.519327 29158 tablet_service.cc:1467] Processing CreateTablet for tablet 2a0ff3bf39984d4c94665ecd3595b412 (DEFAULT_TABLE table=test-table [id=ff35316d9ea54635b3d65d1a57a4d298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:23:40.520658 29293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0ff3bf39984d4c94665ecd3595b412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:40.520750 29023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0ff3bf39984d4c94665ecd3595b412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:40.520750 29158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a0ff3bf39984d4c94665ecd3595b412. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:23:40.533591 29378 tablet_bootstrap.cc:492] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Bootstrap starting.
I20260926 02:23:40.537292 29379 tablet_bootstrap.cc:492] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Bootstrap starting.
I20260926 02:23:40.537822 29378 tablet_bootstrap.cc:654] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:40.539300 29378 log.cc:824] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:40.541669 29380 tablet_bootstrap.cc:492] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Bootstrap starting.
I20260926 02:23:40.541760 29379 tablet_bootstrap.cc:654] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:40.543220 29379 log.cc:824] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:40.544953 29378 tablet_bootstrap.cc:492] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: No bootstrap required, opened a new log
I20260926 02:23:40.545266 29378 ts_tablet_manager.cc:1397] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260926 02:23:40.545439 29380 tablet_bootstrap.cc:654] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Neither blocks nor log segments found. Creating new log.
I20260926 02:23:40.546664 29379 tablet_bootstrap.cc:492] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: No bootstrap required, opened a new log
I20260926 02:23:40.546813 29380 log.cc:824] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Log is configured to *not* fsync() on all Append() calls
I20260926 02:23:40.546967 29379 ts_tablet_manager.cc:1397] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 02:23:40.550154 29380 tablet_bootstrap.cc:492] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: No bootstrap required, opened a new log
I20260926 02:23:40.550455 29380 ts_tablet_manager.cc:1397] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 02:23:40.554275 29378 raft_consensus.cc:348] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.554764 29378 raft_consensus.cc:374] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:40.554863 29378 raft_consensus.cc:729] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f995a6fd37714b8f8ab427d21fa28731, State: Initialized, Role: FOLLOWER
I20260926 02:23:40.555599 29378 consensus_queue.cc:260] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.556804 29359 heartbeater.cc:503] Master 127.12.244.254:36095 was elected leader, sending a full tablet report...
I20260926 02:23:40.557384 29378 ts_tablet_manager.cc:1428] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260926 02:23:40.557682 29379 raft_consensus.cc:348] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.558158 29379 raft_consensus.cc:374] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:40.558262 29379 raft_consensus.cc:729] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56b6a51b92842e4b749c1d74ac1d4e3, State: Initialized, Role: FOLLOWER
I20260926 02:23:40.558975 29379 consensus_queue.cc:260] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.560930 29379 ts_tablet_manager.cc:1428] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 02:23:40.563166 29380 raft_consensus.cc:348] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.563720 29380 raft_consensus.cc:374] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:23:40.563828 29380 raft_consensus.cc:729] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc901c868fb4be8a7a2eeeab3b72059, State: Initialized, Role: FOLLOWER
I20260926 02:23:40.564605 29380 consensus_queue.cc:260] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.566071 29224 heartbeater.cc:503] Master 127.12.244.254:36095 was elected leader, sending a full tablet report...
I20260926 02:23:40.566856 29380 ts_tablet_manager.cc:1428] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260926 02:23:40.576089 29090 tablet.cc:2369] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:40.590960 29385 raft_consensus.cc:482] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:23:40.591323 29385 raft_consensus.cc:504] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.593133 29385 leader_election.cc:288] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f995a6fd37714b8f8ab427d21fa28731 (127.12.244.195:38483), ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775)
I20260926 02:23:40.600540 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.593482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:40775 (local address 127.12.244.193:55705)
0926 02:23:40.594504 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:40.594522 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:40.594702 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:40.595085 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:40.595108 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:40.595134 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:40.595584 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.595594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.596677 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.596682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.598383 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.598395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.599670 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.599677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.599850 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.599874 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:23:40.599954 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:23:40.600059 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":54,"thread_start_us":756,"threads_started":1}
I20260926 02:23:40.600561 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.593597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:48517 (local address 127.12.244.195:38483)
0926 02:23:40.593855 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.593865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.594717 (+ 852us) server_negotiation.cc:407] Connection header received
0926 02:23:40.594966 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.594970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.595105 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.595241 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:40.595916 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.596692 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.598549 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.599420 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.599623 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.599704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.600182 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.600323 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:23:40.600565 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.593758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:55705 (local address 127.12.244.194:40775)
0926 02:23:40.594026 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.594035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.594643 (+ 608us) server_negotiation.cc:407] Connection header received
0926 02:23:40.594867 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.594874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.594954 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.595025 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:23:40.595741 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.596529 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.598563 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.599409 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.599589 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.599696 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.600122 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.600268 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:23:40.600673 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.593515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:38483 (local address 127.12.244.193:48517)
0926 02:23:40.594542 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0926 02:23:40.594557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:23:40.594787 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:23:40.595262 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:23:40.595265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:23:40.595297 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:23:40.595728 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.595740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.596842 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.596845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.598381 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:23:40.598394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.599686 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.599692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:23:40.599889 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.599912 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:23:40.600138 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 02:23:40.600268 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":68,"thread_start_us":568,"threads_started":1}
I20260926 02:23:40.601980 29179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0ff3bf39984d4c94665ecd3595b412" candidate_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" is_pre_election: true
I20260926 02:23:40.602003 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0ff3bf39984d4c94665ecd3595b412" candidate_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f995a6fd37714b8f8ab427d21fa28731" is_pre_election: true
I20260926 02:23:40.602617 29179 raft_consensus.cc:2454] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56b6a51b92842e4b749c1d74ac1d4e3 in term 0.
I20260926 02:23:40.602655 29314 raft_consensus.cc:2454] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56b6a51b92842e4b749c1d74ac1d4e3 in term 0.
I20260926 02:23:40.603389 28979 leader_election.cc:302] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d56b6a51b92842e4b749c1d74ac1d4e3, ecc901c868fb4be8a7a2eeeab3b72059; no voters:
I20260926 02:23:40.603865 29385 raft_consensus.cc:2781] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:23:40.604015 29385 raft_consensus.cc:482] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:23:40.604139 29385 raft_consensus.cc:3037] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:40.606369 29385 raft_consensus.cc:504] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:23:40.607116 29385 leader_election.cc:288] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [CANDIDATE]: Term 1 election: Requested vote from peers f995a6fd37714b8f8ab427d21fa28731 (127.12.244.195:38483), ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775)
I20260926 02:23:40.607816 29179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0ff3bf39984d4c94665ecd3595b412" candidate_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc901c868fb4be8a7a2eeeab3b72059"
I20260926 02:23:40.607810 29314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a0ff3bf39984d4c94665ecd3595b412" candidate_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f995a6fd37714b8f8ab427d21fa28731"
I20260926 02:23:40.608042 29179 raft_consensus.cc:3037] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:40.608065 29314 raft_consensus.cc:3037] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:23:40.610394 29314 raft_consensus.cc:2454] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56b6a51b92842e4b749c1d74ac1d4e3 in term 1.
I20260926 02:23:40.610404 29179 raft_consensus.cc:2454] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56b6a51b92842e4b749c1d74ac1d4e3 in term 1.
I20260926 02:23:40.610931 28976 leader_election.cc:302] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d56b6a51b92842e4b749c1d74ac1d4e3, f995a6fd37714b8f8ab427d21fa28731; no voters:
I20260926 02:23:40.611297 29385 raft_consensus.cc:2781] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:23:40.611838 29385 raft_consensus.cc:686] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 LEADER]: Becoming Leader. State: Replica: d56b6a51b92842e4b749c1d74ac1d4e3, State: Running, Role: LEADER
I20260926 02:23:40.612352 29385 consensus_queue.cc:237] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
W20260926 02:23:40.616474 29225 tablet.cc:2369] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:23:40.617303 28899 catalog_manager.cc:5495] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d56b6a51b92842e4b749c1d74ac1d4e3 (127.12.244.193). New cstate: current_term: 1 leader_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } health_report { overall_health: UNKNOWN } } }
I20260926 02:23:40.640941 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.635231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.12.244.193:38281)
0926 02:23:40.635444 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.635449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.635469 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:23:40.635664 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.635668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.635772 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.635934 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:23:40.636295 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.637038 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.638555 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.638986 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.639156 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.639741 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:23:40.640298 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:23:40.640423 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.640536 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.640611 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:23:40.649919 29314 raft_consensus.cc:1261] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 1 FOLLOWER]: Refusing update from remote peer d56b6a51b92842e4b749c1d74ac1d4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:40.649964 29179 raft_consensus.cc:1261] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Refusing update from remote peer d56b6a51b92842e4b749c1d74ac1d4e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:23:40.650993 29389 consensus_queue.cc:1035] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:40.651459 29385 consensus_queue.cc:1035] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:23:40.667338 29392 mvcc.cc:205] Tried to move back new op lower bound from 7333435066971058176 to 7333435066831888384. Current Snapshot: MvccSnapshot[applied={T|T < 7333435066971058176}]
I20260926 02:23:40.677400 29394 mvcc.cc:205] Tried to move back new op lower bound from 7333435066971058176 to 7333435066831888384. Current Snapshot: MvccSnapshot[applied={T|T < 7333435066971058176}]
I20260926 02:23:40.684198 28899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:40.684506 28899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:23:40.688643 28899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Sending DeleteTablet for 3 replicas of tablet 2a0ff3bf39984d4c94665ecd3595b412
I20260926 02:23:40.689980 29023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0ff3bf39984d4c94665ecd3595b412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:40 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:44268
I20260926 02:23:40.689980 29293 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0ff3bf39984d4c94665ecd3595b412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:40 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:48378
I20260926 02:23:40.690137 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 28960
I20260926 02:23:40.691442 29158 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a0ff3bf39984d4c94665ecd3595b412 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:23:40 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:58344
I20260926 02:23:40.700471 29405 tablet_replica.cc:330] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: stopping tablet replica
I20260926 02:23:40.701172 29405 raft_consensus.cc:2229] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:40.701858 29405 raft_consensus.cc:2258] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:40.709245 29405 ts_tablet_manager.cc:1905] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:40.709317 29402 tablet_replica.cc:330] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: stopping tablet replica
I20260926 02:23:40.709867 29402 raft_consensus.cc:2229] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:23:40.710512 29402 raft_consensus.cc:2258] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:40.711450 29403 tablet_replica.cc:330] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: stopping tablet replica
I20260926 02:23:40.711942 29403 raft_consensus.cc:2229] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:23:40.712685 29403 raft_consensus.cc:2258] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:23:40.713877 29402 ts_tablet_manager.cc:1905] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:40.714924 29403 ts_tablet_manager.cc:1905] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:23:40.721483 29402 ts_tablet_manager.cc:1918] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:40.721709 29402 log.cc:1196] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/2a0ff3bf39984d4c94665ecd3595b412
I20260926 02:23:40.722265 29402 ts_tablet_manager.cc:1939] T 2a0ff3bf39984d4c94665ecd3595b412 P f995a6fd37714b8f8ab427d21fa28731: Deleting consensus metadata
I20260926 02:23:40.723054 29403 ts_tablet_manager.cc:1918] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:40.723273 29403 log.cc:1196] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/2a0ff3bf39984d4c94665ecd3595b412
I20260926 02:23:40.723841 29403 ts_tablet_manager.cc:1939] T 2a0ff3bf39984d4c94665ecd3595b412 P d56b6a51b92842e4b749c1d74ac1d4e3: Deleting consensus metadata
I20260926 02:23:40.724438 28882 catalog_manager.cc:4841] TS f995a6fd37714b8f8ab427d21fa28731 (127.12.244.195:38483): tablet 2a0ff3bf39984d4c94665ecd3595b412 (table test-table [id=ff35316d9ea54635b3d65d1a57a4d298]) successfully deleted
I20260926 02:23:40.725729 28883 catalog_manager.cc:4841] TS d56b6a51b92842e4b749c1d74ac1d4e3 (127.12.244.193:38281): tablet 2a0ff3bf39984d4c94665ecd3595b412 (table test-table [id=ff35316d9ea54635b3d65d1a57a4d298]) successfully deleted
I20260926 02:23:40.727357 29405 ts_tablet_manager.cc:1918] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:23:40.727536 29405 log.cc:1196] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/2a0ff3bf39984d4c94665ecd3595b412
I20260926 02:23:40.728142 29405 ts_tablet_manager.cc:1939] T 2a0ff3bf39984d4c94665ecd3595b412 P ecc901c868fb4be8a7a2eeeab3b72059: Deleting consensus metadata
I20260926 02:23:40.730183 28885 catalog_manager.cc:4841] TS ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775): tablet 2a0ff3bf39984d4c94665ecd3595b412 (table test-table [id=ff35316d9ea54635b3d65d1a57a4d298]) successfully deleted
Sep 26 02:23:40 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:23:40.737953 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.692307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.12.244.193:38281)
0926 02:23:40.692535 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.692543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.714077 (+ 21534us) server_negotiation.cc:407] Connection header received
0926 02:23:40.714636 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.714644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.714735 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.714823 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:40.719298 (+ 4475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.720737 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.724848 (+ 4111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.725610 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.725717 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.731631 (+ 5914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:40.731647 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:40.731662 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:40.731704 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:40.734798 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.735708 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.735716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.735718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.735874 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.736293 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.736297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.736299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.736674 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:40.736998 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.737326 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.737593 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:23:40.738713 29084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:40.840992 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 28960
I20260926 02:23:40.860047 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 29095
Sep 26 02:23:40 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:23:40.879076 29377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.860961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.12.244.194:40775)
0926 02:23:40.861151 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.861157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.861257 (+ 100us) server_negotiation.cc:407] Connection header received
0926 02:23:40.861478 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.861483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.861547 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.861699 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:40.863143 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.863960 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.865621 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.866082 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.866228 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:40.872251 (+ 6023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:40.872267 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:40.872283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:40.872326 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:40.875815 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.876839 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.876847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.876850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.876990 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:40.877481 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:40.877484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:40.877485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:40.877886 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:40.878108 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:40.878574 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 02:23:40.878788 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:23:40.879913 29219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:40.974452 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 29095
I20260926 02:23:40.992266 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 29230
Sep 26 02:23:40 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:23:41.010754 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:23:40.993199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.12.244.195:38483)
0926 02:23:40.993574 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 02:23:40.993581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:23:40.993603 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:23:40.993712 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:23:40.993716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:23:40.993783 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:23:40.993894 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:23:40.995341 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.996166 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.997580 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:23:40.997988 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:23:40.998128 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:23:41.003906 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:23:41.003934 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:23:41.003951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:23:41.004003 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:23:41.007531 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:41.008544 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:41.008553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:41.008555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:41.008723 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:23:41.009167 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:23:41.009171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:23:41.009173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:23:41.009624 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:23:41.009863 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 02:23:41.010247 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 02:23:41.010447 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260926 02:23:41.011623 29354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:41.103578 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 29230
I20260926 02:23:41.121506 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 28866
I20260926 02:23:41.122741 28929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:23:41.236452 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 28866
I20260926 02:24:01.259076 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36095
--webserver_interface=127.12.244.254
--webserver_port=40615
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:01.622359 29410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:01.622767 29410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:01.622964 29410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:01.630851 29410 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:24:01.631049 29410 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:24:01.631178 29410 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:24:01.631274 29410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:24:01.631386 29410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:01.631517 29410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:01.631614 29410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:24:01.631705 29410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:24:01.642609 29410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:36095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:01.644157 29410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:01.645666 29410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:01.670403 29419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:01.670369 29417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:01.670464 29416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:01.672983 29410 server_base.cc:1034] running on GCE node
I20260926 02:24:01.674026 29410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:01.675839 29410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:01.677109 29410 hybrid_clock.cc:648] HybridClock initialized: now 1790389441677021 us; error 65 us; skew 500 ppm
I20260926 02:24:01.677608 29410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:01 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389441, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:01.681448 29410 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:24:01.683581 29410 webserver.cc:460] Webserver started at http://127.12.244.254:40615/ using document root <none> and password file <none>
I20260926 02:24:01.684433 29410 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:01.684604 29410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:01.691255 29410 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 02:24:01.694866 29426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:01.696666 29410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:24:01.696846 29410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "f82c312c1cd34a58998741ec779e2590"
format_stamp: "Formatted at 2026-09-26 02:23:38 on dist-test-slave-qfmw"
server_key: "20eaf7be57909b073eba7b56c54fa43e"
server_key_iv: "b8a35f95ce62f9aa2cb71fcd165ecbb8"
server_key_version: "encryptionkey@0"
I20260926 02:24:01.697968 29410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:01.712352 29410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:01.717378 29410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:01.717656 29410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:01.738165 29410 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:36095
I20260926 02:24:01.738185 29477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:36095 every 8 connection(s)
I20260926 02:24:01.739965 29410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:24:01.745127 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 29410
I20260926 02:24:01.746004 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:38281
--local_ip_for_outbound_sockets=127.12.244.193
--tserver_master_addrs=127.12.244.254:36095
--webserver_port=43173
--webserver_interface=127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:24:01.747839 29478 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:24:01.751487 29478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Bootstrap starting.
I20260926 02:24:01.781524 29478 log.cc:824] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:01.798358 29478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:24:01.799109 29478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Bootstrap complete.
I20260926 02:24:01.813707 29478 raft_consensus.cc:348] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:01.815181 29478 raft_consensus.cc:729] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82c312c1cd34a58998741ec779e2590, State: Initialized, Role: FOLLOWER
I20260926 02:24:01.816162 29478 consensus_queue.cc:260] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:01.816569 29478 raft_consensus.cc:388] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:24:01.816742 29478 raft_consensus.cc:482] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:24:01.816977 29478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 1 FOLLOWER]: Advancing to term 2
I20260926 02:24:01.823736 29478 raft_consensus.cc:504] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:01.824512 29478 leader_election.cc:302] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f82c312c1cd34a58998741ec779e2590; no voters:
I20260926 02:24:01.825335 29478 leader_election.cc:288] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 02:24:01.825595 29482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 02:24:01.827135 29482 raft_consensus.cc:686] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 LEADER]: Becoming Leader. State: Replica: f82c312c1cd34a58998741ec779e2590, State: Running, Role: LEADER
I20260926 02:24:01.827869 29478 sys_catalog.cc:564] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:24:01.827780 29482 consensus_queue.cc:237] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:01.830742 29484 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f82c312c1cd34a58998741ec779e2590. Latest consensus state: current_term: 2 leader_uuid: "f82c312c1cd34a58998741ec779e2590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } } }
I20260926 02:24:01.831234 29484 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:01.830852 29483 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f82c312c1cd34a58998741ec779e2590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } } }
I20260926 02:24:01.831650 29483 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:01.848703 29489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:24:01.857564 29489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:24:01.859766 29489 catalog_manager.cc:1259] Loaded cluster ID: e122588663b64384955ebdbf21444e41
I20260926 02:24:01.859932 29489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:24:01.866708 29489 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:24:01.869143 29489 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Loaded TSK: 0
I20260926 02:24:01.891165 29489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Generated new TSK 1
I20260926 02:24:01.891568 29489 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 02:24:02.168339 29480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:02.168707 29480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:02.168874 29480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:02.176870 29480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:02.177048 29480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:02.177222 29480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:24:02.187889 29480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:38281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=43173
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:02.189302 29480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:02.190766 29480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:02.208312 29505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:02.208631 29508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:02.209978 29480 server_base.cc:1034] running on GCE node
W20260926 02:24:02.211845 29506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:02.212713 29480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:02.213999 29480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:02.215210 29480 hybrid_clock.cc:648] HybridClock initialized: now 1790389442215137 us; error 65 us; skew 500 ppm
I20260926 02:24:02.215639 29480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:02 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389442, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:02.219429 29480 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:24:02.221428 29480 webserver.cc:460] Webserver started at http://127.12.244.193:43173/ using document root <none> and password file <none>
I20260926 02:24:02.222280 29480 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:02.222452 29480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:02.229576 29480 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:24:02.233119 29515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:02.234858 29480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:24:02.235069 29480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "d56b6a51b92842e4b749c1d74ac1d4e3"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "749b2c8ef0fb8e3c8af878a8f3e051e0"
server_key_iv: "4217be066fefbdeac08716c6c1b0814a"
server_key_version: "encryptionkey@0"
I20260926 02:24:02.236289 29480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:02.262641 29480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:02.268095 29480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:02.268435 29480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:02.269629 29480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:02.271304 29480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:02.271447 29480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:02.271595 29480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:02.271703 29480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:02.308861 29480 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:38281
I20260926 02:24:02.308959 29628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:38281 every 8 connection(s)
I20260926 02:24:02.310500 29480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:24:02.316949 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 29480
I20260926 02:24:02.317803 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:40775
--local_ip_for_outbound_sockets=127.12.244.194
--tserver_master_addrs=127.12.244.254:36095
--webserver_port=34253
--webserver_interface=127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:24:02 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389442, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:02.340008 29632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:02.314170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:37187)
0926 02:24:02.315005 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:02.315044 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:02.317286 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:02.319040 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:02.319055 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:02.319922 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:02.320775 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:02.320804 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.322709 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.322717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:02.324925 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:02.324946 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.326429 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.326435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:02.326649 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:02.327437 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:02.327466 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:02.330077 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:02.335253 (+ 5176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:02.335262 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:02.335278 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:02.335830 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:02.336420 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:02.336424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:02.336426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:02.336660 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:02.337771 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:02.337778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:02.338133 (+ 355us) client_negotiation.cc:769] Sending connection context
0926 02:24:02.338371 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 02:24:02.338802 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":354,"threads_started":1}
I20260926 02:24:02.340665 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:02.313755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:37187 (local address 127.12.244.254:36095)
0926 02:24:02.314379 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:02.314398 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:02.315232 (+ 834us) server_negotiation.cc:407] Connection header received
0926 02:24:02.317994 (+ 2762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:02.318023 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:02.318464 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:02.318807 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:02.321053 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.322496 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.325087 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.326168 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.326371 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:02.330451 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:02.330465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:02.330491 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:02.330560 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:02.334936 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:02.336019 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:02.336032 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:02.336042 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:02.336294 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:02.336798 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:02.336806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:02.336812 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:02.337605 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:02.337831 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:02.338689 (+ 858us) server_negotiation.cc:299] Negotiation successful
0926 02:24:02.339244 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":234,"threads_started":1}
I20260926 02:24:02.343827 29629 heartbeater.cc:347] Connected to a master server at 127.12.244.254:36095
I20260926 02:24:02.344214 29629 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:02.345273 29629 heartbeater.cc:511] Master 127.12.244.254:36095 requested a full tablet report, sending...
I20260926 02:24:02.348137 29443 ts_manager.cc:194] Registered new tserver with Master: d56b6a51b92842e4b749c1d74ac1d4e3 (127.12.244.193:38281)
I20260926 02:24:02.350186 29443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:37187
W20260926 02:24:02.673130 29633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:02.673452 29633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:02.673547 29633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:02.680994 29633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:02.681141 29633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:02.681260 29633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:24:02.691421 29633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:40775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=34253
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:02.692808 29633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:02.694347 29633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:02.713624 29639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:02.714275 29640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:02.714905 29642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:02.715410 29633 server_base.cc:1034] running on GCE node
I20260926 02:24:02.716225 29633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:02.717509 29633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:02.718745 29633 hybrid_clock.cc:648] HybridClock initialized: now 1790389442718670 us; error 78 us; skew 500 ppm
I20260926 02:24:02.719156 29633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:02 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389442, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:02.723392 29633 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:24:02.725095 29633 webserver.cc:460] Webserver started at http://127.12.244.194:34253/ using document root <none> and password file <none>
I20260926 02:24:02.725893 29633 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:02.726049 29633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:02.732276 29633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:24:02.735525 29649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:02.737041 29633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:24:02.737288 29633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "ecc901c868fb4be8a7a2eeeab3b72059"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "43ce1f0d57a4962bfb5f314bfa37b45b"
server_key_iv: "5a459a58191cadf686f64b1d6bde9af4"
server_key_version: "encryptionkey@0"
I20260926 02:24:02.738322 29633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:02.758342 29633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:02.762876 29633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:02.763221 29633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:02.764398 29633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:02.765908 29633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:02.766002 29633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:02.766177 29633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:02.766247 29633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:02.801836 29633 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:40775
I20260926 02:24:02.801921 29761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:40775 every 8 connection(s)
I20260926 02:24:02.803534 29633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:24:02.806917 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 29633
I20260926 02:24:02.807780 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:38483
--local_ip_for_outbound_sockets=127.12.244.195
--tserver_master_addrs=127.12.244.254:36095
--webserver_port=39281
--webserver_interface=127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 02:24:02 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389442, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:02.830130 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:02.806371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:53953 (local address 127.12.244.254:36095)
0926 02:24:02.806667 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:02.806676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:02.808046 (+ 1370us) server_negotiation.cc:407] Connection header received
0926 02:24:02.811222 (+ 3176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:02.811227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:02.811321 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:02.811413 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:02.813823 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.814601 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.816974 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.817430 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.817575 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:02.821757 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:02.821766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:02.821774 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:02.821829 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:02.825803 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:02.826795 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:02.826804 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:02.826811 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:02.826994 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:02.827581 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:02.827588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:02.827595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:02.828088 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:02.828323 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:02.829295 (+ 972us) server_negotiation.cc:299] Negotiation successful
0926 02:24:02.829636 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 02:24:02.830858 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:02.806917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:53953)
0926 02:24:02.807853 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:02.807909 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:02.810820 (+ 2911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:02.811740 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:02.811754 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:02.812646 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:02.813562 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:02.813602 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.814793 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.814801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:02.816799 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:02.816813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:02.817685 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:02.817693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:02.817957 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:02.818756 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:02.818781 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:02.821429 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:02.826030 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:02.826039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:02.826056 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:02.826613 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:02.827195 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:02.827198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:02.827201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:02.827438 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:02.828332 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:02.828341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:02.828797 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 02:24:02.829111 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 02:24:02.829649 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":355,"threads_started":1}
I20260926 02:24:02.832669 29762 heartbeater.cc:347] Connected to a master server at 127.12.244.254:36095
I20260926 02:24:02.833062 29762 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:02.834143 29762 heartbeater.cc:511] Master 127.12.244.254:36095 requested a full tablet report, sending...
I20260926 02:24:02.836182 29443 ts_manager.cc:194] Registered new tserver with Master: ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775)
I20260926 02:24:02.837102 29443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:53953
W20260926 02:24:03.170596 29765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:03.170938 29765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:03.171139 29765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:03.178861 29765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:03.179008 29765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:03.179241 29765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:24:03.190351 29765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:38483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=39281
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:03.191846 29765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:03.193275 29765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:03.208041 29771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:03.208099 29773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:03.208104 29770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:03.208948 29765 server_base.cc:1034] running on GCE node
I20260926 02:24:03.209671 29765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:03.210711 29765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:03.211944 29765 hybrid_clock.cc:648] HybridClock initialized: now 1790389443211874 us; error 67 us; skew 500 ppm
I20260926 02:24:03.212296 29765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:03 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389443, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:03.215773 29765 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:24:03.217435 29765 webserver.cc:460] Webserver started at http://127.12.244.195:39281/ using document root <none> and password file <none>
I20260926 02:24:03.218211 29765 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:03.218374 29765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:03.224515 29765 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:24:03.227625 29780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:03.229365 29765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 02:24:03.229606 29765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "f995a6fd37714b8f8ab427d21fa28731"
format_stamp: "Formatted at 2026-09-26 02:23:40 on dist-test-slave-qfmw"
server_key: "e83137734c93ecb4d0e5438ba63840e6"
server_key_iv: "a1413a3a17036a62fb422640b89bb879"
server_key_version: "encryptionkey@0"
I20260926 02:24:03.230706 29765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:03.247833 29765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:03.252655 29765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:03.252959 29765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:03.254896 29765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:03.256537 29765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:03.256672 29765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:03.256790 29765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:03.256892 29765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:03.291601 29765 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:38483
I20260926 02:24:03.291735 29894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:38483 every 8 connection(s)
I20260926 02:24:03.293237 29765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:24:03.298116 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 29765
Sep 26 02:24:03 dist-test-slave-qfmw krb5kdc[28853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389443, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:03.316001 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.299274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.12.244.254:36095)
0926 02:24:03.300338 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.300342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.300358 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:24:03.300410 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.300413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.300471 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.300592 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:03.302257 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.303554 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.305002 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.305480 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.305591 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.308975 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:03.308980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:03.308983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:03.309017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:03.312477 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.313490 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.313498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.313502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.313679 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.314156 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.314163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.314167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.314598 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:03.314900 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.315238 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.315484 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":964,"thread_start_us":849,"threads_started":1}
I20260926 02:24:03.319845 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.296156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:39593 (local address 127.12.244.254:36095)
0926 02:24:03.296336 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.296340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.297721 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 02:24:03.300490 (+ 2769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.300495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.300571 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.300685 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:03.302383 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.303554 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.307233 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.307681 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.307780 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.313015 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:03.313019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:03.313023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:03.313055 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:03.316356 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.317327 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.317331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.317333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.317463 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.318034 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.318038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.318041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.318335 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:03.318518 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.319387 (+ 869us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.319566 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":61}
I20260926 02:24:03.320794 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.296666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:39593)
0926 02:24:03.297451 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:03.297497 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:03.300189 (+ 2692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:03.300951 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:03.300960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:03.301550 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:03.302202 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.302220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.305043 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.305047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.307043 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.307063 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.307956 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.307965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.308314 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.309689 (+ 1375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:03.309744 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:03.312785 (+ 3041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:03.316582 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:03.316597 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:03.316612 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:03.317181 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:03.317581 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:03.317594 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:03.317597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:03.317888 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:03.318482 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:03.318491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:03.318971 (+ 480us) client_negotiation.cc:769] Sending connection context
0926 02:24:03.319261 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 02:24:03.319679 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":320,"threads_started":1}
I20260926 02:24:03.322264 29895 heartbeater.cc:347] Connected to a master server at 127.12.244.254:36095
I20260926 02:24:03.322618 29895 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:03.323465 29895 heartbeater.cc:511] Master 127.12.244.254:36095 requested a full tablet report, sending...
I20260926 02:24:03.324994 29443 ts_manager.cc:194] Registered new tserver with Master: f995a6fd37714b8f8ab427d21fa28731 (127.12.244.195:38483)
I20260926 02:24:03.325907 29443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:39593
I20260926 02:24:03.331494 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:24:03.338320 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.332535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.12.244.254:36095)
0926 02:24:03.332696 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.332700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.333129 (+ 429us) server_negotiation.cc:407] Connection header received
0926 02:24:03.333338 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.333342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.333486 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.333681 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:24:03.334057 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.334871 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.336509 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.336980 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.337090 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.337592 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:24:03.337821 (+ 229us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 02:24:03.338113 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40676: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 02:24:03.338428 29631 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40676: token expired
W20260926 02:24:03.339175 29366 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.244.254:36095)
I20260926 02:24:03.353623 29631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.339819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.12.244.254:36095)
0926 02:24:03.340007 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.340012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.340026 (+ 14us) server_negotiation.cc:407] Connection header received
0926 02:24:03.340064 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.340067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.340136 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.340233 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:03.341875 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.342715 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.344243 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.344668 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.344841 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.347831 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:03.347836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:03.347839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:03.347882 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:03.350844 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.351721 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.351725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.351728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.351871 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.352252 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.352255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.352257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.352624 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:03.352819 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.353163 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.353359 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:24:03.354097 29629 heartbeater.cc:503] Master 127.12.244.254:36095 was elected leader, sending a full tablet report...
I20260926 02:24:03.357760 29443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:24:03.365569 29443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:24:03.389938 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.378653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.12.244.195:38483)
0926 02:24:03.379643 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.379655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.380371 (+ 716us) server_negotiation.cc:407] Connection header received
0926 02:24:03.380726 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.380737 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.381002 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.381189 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:03.382917 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.384519 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.387017 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.388560 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.388778 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.388974 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.389342 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.389554 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":652,"threads_started":1}
I20260926 02:24:03.390058 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.379266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.12.244.193:38281)
0926 02:24:03.381071 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.381079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.381107 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:24:03.381207 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.381214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.381429 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.382416 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:03.382652 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.384332 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.387199 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.388612 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.388799 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.388974 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.389551 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.389734 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1624,"thread_start_us":1497,"threads_started":1}
I20260926 02:24:03.390174 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.378421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:38281 (local address 127.0.0.1:35474)
0926 02:24:03.380212 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:03.380259 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:03.380448 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:03.381624 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:03.381630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:03.381655 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:03.382179 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.382192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.384485 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.384491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.387025 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.387040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.389162 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.389171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.389349 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.389379 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:24:03.389476 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:24:03.389703 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1593,"mutex_wait_us":68,"spinlock_wait_cycles":11520,"thread_start_us":1327,"threads_started":1}
I20260926 02:24:03.390893 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.378347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:38483 (local address 127.0.0.1:52186)
0926 02:24:03.380212 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:03.380259 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:03.380463 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:03.381553 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:03.381562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:03.381603 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:03.382697 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.382718 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.384657 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.384663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.386839 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.386858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.388802 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.388808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.389019 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.389093 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 02:24:03.390328 (+ 1235us) client_negotiation.cc:240] Negotiation successful
0926 02:24:03.390470 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1571,"mutex_wait_us":551,"thread_start_us":2105,"threads_started":1}
I20260926 02:24:03.391549 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.379836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:40775 (local address 127.0.0.1:41144)
0926 02:24:03.382726 (+ 2890us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:03.382744 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:03.382870 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:03.383645 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:03.383650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:03.383674 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:03.383986 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.383994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.385801 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.385806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.387714 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.387725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.390577 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.390585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.390774 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.390805 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 02:24:03.390918 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:24:03.391168 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2760,"mutex_wait_us":1,"thread_start_us":2598,"threads_started":1}
I20260926 02:24:03.391544 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.380406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.12.244.194:40775)
0926 02:24:03.382370 (+ 1964us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.382379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.382999 (+ 620us) server_negotiation.cc:407] Connection header received
0926 02:24:03.383132 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.383140 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.383354 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.383541 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:03.384144 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.385621 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.387873 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.389026 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.389201 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.389386 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.390989 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.391217 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":1649,"threads_started":1}
I20260926 02:24:03.394902 29697 tablet_service.cc:1467] Processing CreateTablet for tablet d04fa1517a1441969cd03b0238397f17 (DEFAULT_TABLE table=test-table [id=78333ff7db86470f9c1ea50ca8434e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:03.394902 29830 tablet_service.cc:1467] Processing CreateTablet for tablet d04fa1517a1441969cd03b0238397f17 (DEFAULT_TABLE table=test-table [id=78333ff7db86470f9c1ea50ca8434e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:03.394902 29564 tablet_service.cc:1467] Processing CreateTablet for tablet d04fa1517a1441969cd03b0238397f17 (DEFAULT_TABLE table=test-table [id=78333ff7db86470f9c1ea50ca8434e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:03.396284 29697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04fa1517a1441969cd03b0238397f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:03.396400 29830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04fa1517a1441969cd03b0238397f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:03.396420 29564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d04fa1517a1441969cd03b0238397f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:03.410378 29911 tablet_bootstrap.cc:492] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Bootstrap starting.
I20260926 02:24:03.414769 29913 tablet_bootstrap.cc:492] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Bootstrap starting.
I20260926 02:24:03.415030 29912 tablet_bootstrap.cc:492] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Bootstrap starting.
I20260926 02:24:03.415309 29911 tablet_bootstrap.cc:654] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:03.416724 29911 log.cc:824] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:03.418288 29913 tablet_bootstrap.cc:654] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:03.418385 29912 tablet_bootstrap.cc:654] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:03.419519 29912 log.cc:824] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:03.419603 29913 log.cc:824] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:03.420459 29911 tablet_bootstrap.cc:492] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: No bootstrap required, opened a new log
I20260926 02:24:03.420737 29911 ts_tablet_manager.cc:1397] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 02:24:03.422847 29913 tablet_bootstrap.cc:492] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: No bootstrap required, opened a new log
I20260926 02:24:03.422855 29912 tablet_bootstrap.cc:492] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: No bootstrap required, opened a new log
I20260926 02:24:03.423154 29912 ts_tablet_manager.cc:1397] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 02:24:03.423163 29913 ts_tablet_manager.cc:1397] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 02:24:03.430512 29913 raft_consensus.cc:348] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.430950 29913 raft_consensus.cc:374] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:03.431052 29913 raft_consensus.cc:729] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f995a6fd37714b8f8ab427d21fa28731, State: Initialized, Role: FOLLOWER
I20260926 02:24:03.431787 29913 consensus_queue.cc:260] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.431795 29911 raft_consensus.cc:348] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.432312 29911 raft_consensus.cc:374] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:03.432418 29911 raft_consensus.cc:729] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc901c868fb4be8a7a2eeeab3b72059, State: Initialized, Role: FOLLOWER
I20260926 02:24:03.433156 29911 consensus_queue.cc:260] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.433851 29913 ts_tablet_manager.cc:1428] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260926 02:24:03.434343 29895 heartbeater.cc:503] Master 127.12.244.254:36095 was elected leader, sending a full tablet report...
I20260926 02:24:03.434218 29912 raft_consensus.cc:348] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.434695 29912 raft_consensus.cc:374] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:03.434804 29912 raft_consensus.cc:729] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56b6a51b92842e4b749c1d74ac1d4e3, State: Initialized, Role: FOLLOWER
I20260926 02:24:03.435297 29762 heartbeater.cc:503] Master 127.12.244.254:36095 was elected leader, sending a full tablet report...
I20260926 02:24:03.435403 29912 consensus_queue.cc:260] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.438282 29912 ts_tablet_manager.cc:1428] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 02:24:03.440011 29911 ts_tablet_manager.cc:1428] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Time spent starting tablet: real 0.019s user 0.010s sys 0.005s
I20260926 02:24:03.519865 29917 raft_consensus.cc:482] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:24:03.520205 29917 raft_consensus.cc:504] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.521991 29917 leader_election.cc:288] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d56b6a51b92842e4b749c1d74ac1d4e3 (127.12.244.193:38281), ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775)
I20260926 02:24:03.527796 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.522574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:48695 (local address 127.12.244.194:40775)
0926 02:24:03.522771 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.522775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.522814 (+ 39us) server_negotiation.cc:407] Connection header received
0926 02:24:03.522881 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.522885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.522942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.523015 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:03.523539 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.524521 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.526116 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.526881 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.527038 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.527131 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.527412 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.527558 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:24:03.527791 29897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.522298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:40775 (local address 127.12.244.195:48695)
0926 02:24:03.522607 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:03.522627 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:03.522754 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:03.523046 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:03.523050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:03.523071 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:03.523314 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.523322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.524643 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.524651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.525931 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.525943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.527034 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.527039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.527241 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.527267 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 02:24:03.527372 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:24:03.527503 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 02:24:03.528890 29920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.522455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:38281 (local address 127.12.244.195:54647)
0926 02:24:03.523839 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:03.523855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:03.523948 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:03.524232 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:03.524235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:03.524251 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:03.524516 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.524523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.525517 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.525523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.526899 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:03.526911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.528151 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.528159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:03.528328 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.528369 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:24:03.528474 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:24:03.528611 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"mutex_wait_us":689,"thread_start_us":384,"threads_started":1}
I20260926 02:24:03.528909 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.522393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:54647 (local address 127.12.244.193:38281)
0926 02:24:03.522610 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.522617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.523942 (+ 1325us) server_negotiation.cc:407] Connection header received
0926 02:24:03.524035 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.524041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.524116 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.524204 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:03.524681 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.525382 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.527047 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.527908 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.528101 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.528166 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.528509 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.528658 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:24:03.529034 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04fa1517a1441969cd03b0238397f17" candidate_uuid: "f995a6fd37714b8f8ab427d21fa28731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" is_pre_election: true
I20260926 02:24:03.529619 29717 raft_consensus.cc:2454] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f995a6fd37714b8f8ab427d21fa28731 in term 0.
I20260926 02:24:03.530011 29584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04fa1517a1441969cd03b0238397f17" candidate_uuid: "f995a6fd37714b8f8ab427d21fa28731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" is_pre_election: true
I20260926 02:24:03.530309 29784 leader_election.cc:302] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ecc901c868fb4be8a7a2eeeab3b72059, f995a6fd37714b8f8ab427d21fa28731; no voters:
I20260926 02:24:03.530591 29584 raft_consensus.cc:2454] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f995a6fd37714b8f8ab427d21fa28731 in term 0.
I20260926 02:24:03.530777 29917 raft_consensus.cc:2781] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:24:03.530928 29917 raft_consensus.cc:482] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:24:03.531050 29917 raft_consensus.cc:3037] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:03.533313 29917 raft_consensus.cc:504] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.533990 29917 leader_election.cc:288] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [CANDIDATE]: Term 1 election: Requested vote from peers d56b6a51b92842e4b749c1d74ac1d4e3 (127.12.244.193:38281), ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775)
I20260926 02:24:03.534427 29584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04fa1517a1441969cd03b0238397f17" candidate_uuid: "f995a6fd37714b8f8ab427d21fa28731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3"
I20260926 02:24:03.534487 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d04fa1517a1441969cd03b0238397f17" candidate_uuid: "f995a6fd37714b8f8ab427d21fa28731" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc901c868fb4be8a7a2eeeab3b72059"
I20260926 02:24:03.534647 29584 raft_consensus.cc:3037] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:03.534713 29717 raft_consensus.cc:3037] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:03.537097 29717 raft_consensus.cc:2454] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f995a6fd37714b8f8ab427d21fa28731 in term 1.
I20260926 02:24:03.537212 29584 raft_consensus.cc:2454] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f995a6fd37714b8f8ab427d21fa28731 in term 1.
I20260926 02:24:03.537596 29784 leader_election.cc:302] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ecc901c868fb4be8a7a2eeeab3b72059, f995a6fd37714b8f8ab427d21fa28731; no voters:
I20260926 02:24:03.537974 29917 raft_consensus.cc:2781] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:03.538517 29917 raft_consensus.cc:686] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 1 LEADER]: Becoming Leader. State: Replica: f995a6fd37714b8f8ab427d21fa28731, State: Running, Role: LEADER
I20260926 02:24:03.539081 29917 consensus_queue.cc:237] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } }
I20260926 02:24:03.544155 29441 catalog_manager.cc:5495] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 reported cstate change: term changed from 0 to 1, leader changed from <none> to f995a6fd37714b8f8ab427d21fa28731 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "f995a6fd37714b8f8ab427d21fa28731" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f995a6fd37714b8f8ab427d21fa28731" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 38483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 } health_report { overall_health: UNKNOWN } } }
W20260926 02:24:03.544922 29896 tablet.cc:2369] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:03.555822 29763 tablet.cc:2369] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:03.563167 29630 tablet.cc:2369] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:24:03.588080 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.582556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.12.244.195:38483)
0926 02:24:03.582770 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.582776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.582886 (+ 110us) server_negotiation.cc:407] Connection header received
0926 02:24:03.583148 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.583153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.583231 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.583384 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:24:03.583738 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.584452 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.585892 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.586294 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.586425 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.586933 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:24:03.587499 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:24:03.587676 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.587729 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.587812 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:24:03.596958 29717 raft_consensus.cc:1261] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Refusing update from remote peer f995a6fd37714b8f8ab427d21fa28731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:03.596958 29584 raft_consensus.cc:1261] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 FOLLOWER]: Refusing update from remote peer f995a6fd37714b8f8ab427d21fa28731: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:03.598246 29917 consensus_queue.cc:1035] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d56b6a51b92842e4b749c1d74ac1d4e3" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 38281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:03.598628 29921 consensus_queue.cc:1035] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc901c868fb4be8a7a2eeeab3b72059" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 40775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:03.614828 29925 mvcc.cc:205] Tried to move back new op lower bound from 7333435160961069056 to 7333435160740057088. Current Snapshot: MvccSnapshot[applied={T|T < 7333435160961069056}]
I20260926 02:24:03.632691 29441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:03.632995 29441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:03.636726 29441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Sending DeleteTablet for 3 replicas of tablet d04fa1517a1441969cd03b0238397f17
I20260926 02:24:03.638149 29564 tablet_service.cc:1514] Processing DeleteTablet for tablet d04fa1517a1441969cd03b0238397f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:03 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:35474
I20260926 02:24:03.638166 29830 tablet_service.cc:1514] Processing DeleteTablet for tablet d04fa1517a1441969cd03b0238397f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:03 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:52186
I20260926 02:24:03.638502 29697 tablet_service.cc:1514] Processing DeleteTablet for tablet d04fa1517a1441969cd03b0238397f17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:03 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:41144
I20260926 02:24:03.639372 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 29480
I20260926 02:24:03.650731 29936 tablet_replica.cc:330] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: stopping tablet replica
I20260926 02:24:03.651492 29936 raft_consensus.cc:2229] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:03.652238 29936 raft_consensus.cc:2258] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:03.654865 29936 ts_tablet_manager.cc:1905] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:03.668141 29936 ts_tablet_manager.cc:1918] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:03.668437 29936 log.cc:1196] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/d04fa1517a1441969cd03b0238397f17
I20260926 02:24:03.669164 29936 ts_tablet_manager.cc:1939] T d04fa1517a1441969cd03b0238397f17 P d56b6a51b92842e4b749c1d74ac1d4e3: Deleting consensus metadata
I20260926 02:24:03.670921 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.640717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.12.244.193:38281)
0926 02:24:03.640952 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.640960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.640984 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:24:03.641056 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.641065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.641180 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.641310 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:03.644275 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.645927 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.650911 (+ 4984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.651609 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.651715 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.660185 (+ 8470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:03.660201 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:03.660221 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:03.660276 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:03.665381 (+ 5105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.666700 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.666715 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.666721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.667008 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.668005 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.668009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.668012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.668550 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:03.668905 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.670262 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.670523 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 02:24:03.671674 29624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:03.674064 29935 tablet_replica.cc:330] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: stopping tablet replica
I20260926 02:24:03.674724 29935 raft_consensus.cc:2229] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:24:03.675645 29935 raft_consensus.cc:2258] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:03.678000 29428 catalog_manager.cc:4841] TS d56b6a51b92842e4b749c1d74ac1d4e3 (127.12.244.193:38281): tablet d04fa1517a1441969cd03b0238397f17 (table test-table [id=78333ff7db86470f9c1ea50ca8434e47]) successfully deleted
I20260926 02:24:03.678628 29935 ts_tablet_manager.cc:1905] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:03.685119 29937 tablet_replica.cc:330] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: stopping tablet replica
I20260926 02:24:03.685874 29937 raft_consensus.cc:2229] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:03.686517 29937 raft_consensus.cc:2258] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:03.689054 29937 ts_tablet_manager.cc:1905] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:03.691226 29935 ts_tablet_manager.cc:1918] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:03.691560 29935 log.cc:1196] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/d04fa1517a1441969cd03b0238397f17
I20260926 02:24:03.692204 29935 ts_tablet_manager.cc:1939] T d04fa1517a1441969cd03b0238397f17 P f995a6fd37714b8f8ab427d21fa28731: Deleting consensus metadata
I20260926 02:24:03.694409 29427 catalog_manager.cc:4841] TS f995a6fd37714b8f8ab427d21fa28731 (127.12.244.195:38483): tablet d04fa1517a1441969cd03b0238397f17 (table test-table [id=78333ff7db86470f9c1ea50ca8434e47]) successfully deleted
I20260926 02:24:03.696090 29937 ts_tablet_manager.cc:1918] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:03.696290 29937 log.cc:1196] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/d04fa1517a1441969cd03b0238397f17
I20260926 02:24:03.696770 29937 ts_tablet_manager.cc:1939] T d04fa1517a1441969cd03b0238397f17 P ecc901c868fb4be8a7a2eeeab3b72059: Deleting consensus metadata
I20260926 02:24:03.698460 29430 catalog_manager.cc:4841] TS ecc901c868fb4be8a7a2eeeab3b72059 (127.12.244.194:40775): tablet d04fa1517a1441969cd03b0238397f17 (table test-table [id=78333ff7db86470f9c1ea50ca8434e47]) successfully deleted
I20260926 02:24:03.829378 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 29480
I20260926 02:24:03.849294 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 29633
I20260926 02:24:03.866801 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.850297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.12.244.194:40775)
0926 02:24:03.850493 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.850498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.850519 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:24:03.850730 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.850735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.850804 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.850897 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:03.852831 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.853632 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.855738 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.856181 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.856299 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:03.860110 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:03.860125 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:03.860142 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:03.860187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:03.863405 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.864347 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.864371 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.864374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.864584 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:03.865009 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:03.865012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:03.865014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:03.865399 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:03.865685 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:03.866142 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 02:24:03.866420 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 02:24:03.867626 29757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:03.968478 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 29633
I20260926 02:24:03.992640 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 29765
I20260926 02:24:04.010133 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:03.993513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.12.244.195:38483)
0926 02:24:03.993717 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:03.993722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:03.993742 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:24:03.993988 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:03.993993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:03.994090 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:03.994183 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:03.996230 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.997118 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.999180 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:03.999631 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:03.999737 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:04.003442 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:04.003459 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:04.003475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:04.003530 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:04.006735 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:04.007884 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:04.007893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:04.007895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:04.008086 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:04.008531 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:04.008536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:04.008539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:04.008933 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:04.009234 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:04.009594 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 02:24:04.009771 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:24:04.010982 29890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:04.137326 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 29765
I20260926 02:24:04.157526 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 29410
I20260926 02:24:04.158795 29473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:04.274866 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 29410
I20260926 02:24:24.297996 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36095
--webserver_interface=127.12.244.254
--webserver_port=40615
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:36095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:24.660492 29942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:24.660881 29942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:24.661053 29942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:24.668762 29942 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 02:24:24.668931 29942 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 02:24:24.669028 29942 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 02:24:24.669129 29942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:24:24.669239 29942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:24.669368 29942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:24.669458 29942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:24:24.669625 29942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:24:24.680150 29942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:36095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=40615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:24.681659 29942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:24.683192 29942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:24.697374 29950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:24.697440 29948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:24.697453 29947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:24.699255 29942 server_base.cc:1034] running on GCE node
I20260926 02:24:24.700471 29942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:24.702584 29942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:24.703845 29942 hybrid_clock.cc:648] HybridClock initialized: now 1790389464703754 us; error 63 us; skew 500 ppm
I20260926 02:24:24.704276 29942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:24 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389464, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:24.708698 29942 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:24:24.710705 29942 webserver.cc:460] Webserver started at http://127.12.244.254:40615/ using document root <none> and password file <none>
I20260926 02:24:24.711585 29942 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:24.711714 29942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:24.718037 29942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 02:24:24.721580 29957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:24.723071 29942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:24:24.723347 29942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "f82c312c1cd34a58998741ec779e2590"
format_stamp: "Formatted at 2026-09-26 02:23:38 on dist-test-slave-qfmw"
server_key: "20eaf7be57909b073eba7b56c54fa43e"
server_key_iv: "b8a35f95ce62f9aa2cb71fcd165ecbb8"
server_key_version: "encryptionkey@0"
I20260926 02:24:24.724444 29942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:24.744721 29942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:24.749718 29942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:24.750046 29942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:24.770927 29942 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:36095
I20260926 02:24:24.770959 30009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:36095 every 8 connection(s)
I20260926 02:24:24.772684 29942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:24:24.774528 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 29942
I20260926 02:24:24.775636 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:38281
--local_ip_for_outbound_sockets=127.12.244.193
--tserver_master_addrs=127.12.244.254:36095
--webserver_port=43173
--webserver_interface=127.12.244.193
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:24:24.783711 30010 sys_catalog.cc:263] Verifying existing consensus state
I20260926 02:24:24.787709 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Bootstrap starting.
I20260926 02:24:24.809563 30010 log.cc:824] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:24.827394 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 02:24:24.828047 30010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Bootstrap complete.
I20260926 02:24:24.838150 30010 raft_consensus.cc:348] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:24.839169 30010 raft_consensus.cc:729] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f82c312c1cd34a58998741ec779e2590, State: Initialized, Role: FOLLOWER
I20260926 02:24:24.839839 30010 consensus_queue.cc:260] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:24.840135 30010 raft_consensus.cc:388] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:24:24.840276 30010 raft_consensus.cc:482] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:24:24.840502 30010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 2 FOLLOWER]: Advancing to term 3
I20260926 02:24:24.844887 30010 raft_consensus.cc:504] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:24.845528 30010 leader_election.cc:302] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f82c312c1cd34a58998741ec779e2590; no voters:
I20260926 02:24:24.846320 30010 leader_election.cc:288] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 02:24:24.846464 30014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 02:24:24.853528 30010 sys_catalog.cc:564] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:24:24.853616 30014 raft_consensus.cc:686] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [term 3 LEADER]: Becoming Leader. State: Replica: f82c312c1cd34a58998741ec779e2590, State: Running, Role: LEADER
I20260926 02:24:24.854382 30014 consensus_queue.cc:237] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } }
I20260926 02:24:24.857808 30015 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f82c312c1cd34a58998741ec779e2590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } } }
I20260926 02:24:24.858305 30015 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:24.859340 30022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:24:24.864468 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f82c312c1cd34a58998741ec779e2590. Latest consensus state: current_term: 3 leader_uuid: "f82c312c1cd34a58998741ec779e2590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f82c312c1cd34a58998741ec779e2590" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36095 } } }
I20260926 02:24:24.865058 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590 [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:24.875168 30022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:24:24.876796 30022 catalog_manager.cc:1259] Loaded cluster ID: e122588663b64384955ebdbf21444e41
I20260926 02:24:24.877002 30022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:24:24.881258 30022 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:24:24.883123 30022 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Loaded TSK: 0
I20260926 02:24:24.883420 30022 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f82c312c1cd34a58998741ec779e2590: Loaded TSK: 1
F20260926 02:24:24.884790 30022 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f60b75ca9ad google::LogMessage::Fail() at ??:0
@ 0x7f60b75cd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f60b75ca4de google::LogMessage::Flush() at ??:0
@ 0x7f60b75cba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f60bb32e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f60bb31d73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f60ccff91c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f60ccff8783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f60cd0842a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
I20260926 02:24:24.942031 30037 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f60cd0840c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f60cd0ce5e9 std::function<>::operator()() at ??:0
@ 0x7f60ccffccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f60ccffac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f60cd082610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f60cd082439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f60ba216345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f60ba21991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f60ba219799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f60ba1f4eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f60bd4ce6db start_thread at ??:0
@ 0x7f60b646461f clone at ??:0
W20260926 02:24:25.214413 30012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:25.214726 30012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:25.214892 30012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:25.222448 30012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:25.222590 30012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:25.222790 30012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:24:25.233318 30012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:38281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=43173
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:25.234766 30012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:25.236191 30012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:25.250029 30059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:25.250128 30062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:25.250144 30060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:25.250730 30012 server_base.cc:1034] running on GCE node
I20260926 02:24:25.251510 30012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:25.252564 30012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:25.253782 30012 hybrid_clock.cc:648] HybridClock initialized: now 1790389465253726 us; error 53 us; skew 500 ppm
I20260926 02:24:25.254171 30012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:25 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:25.257931 30012 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:24:25.259966 30012 webserver.cc:460] Webserver started at http://127.12.244.193:43173/ using document root <none> and password file <none>
I20260926 02:24:25.260763 30012 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:25.260936 30012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:25.267223 30012 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 02:24:25.272281 30069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:25.274124 30012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:24:25.274348 30012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "d56b6a51b92842e4b749c1d74ac1d4e3"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "749b2c8ef0fb8e3c8af878a8f3e051e0"
server_key_iv: "4217be066fefbdeac08716c6c1b0814a"
server_key_version: "encryptionkey@0"
I20260926 02:24:25.275552 30012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:25.303282 30012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:25.308620 30012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:25.308985 30012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:25.310163 30012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:25.311949 30012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:25.312093 30012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:25.312270 30012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:25.312372 30012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:25.352367 30012 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:38281
I20260926 02:24:25.352465 30182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:38281 every 8 connection(s)
I20260926 02:24:25.354172 30012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:24:25.356459 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 30012
I20260926 02:24:25.357362 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:40775
--local_ip_for_outbound_sockets=127.12.244.194
--tserver_master_addrs=127.12.244.254:36095
--webserver_port=34253
--webserver_interface=127.12.244.194
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:24:25.361065 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:25.358420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:59417)
0926 02:24:25.359470 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:25.359761 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":407,"threads_started":1}
W20260926 02:24:25.361876 30183 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:36095 (0 consecutive failures): Network error: Failed to ping master at 127.12.244.254:36095: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
I20260926 02:24:25.363121 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:25.362551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:40415)
0926 02:24:25.362789 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:25.362902 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 02:24:25.364324 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:25.363816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:53789)
0926 02:24:25.364018 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:25.364139 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:24:25.364641 30183 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:24:25.715831 30186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:25.716140 30186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:25.716238 30186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:25.723750 30186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:25.723878 30186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:25.723996 30186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:24:25.734380 30186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:40775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=34253
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:25.735839 30186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:25.737195 30186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:25.751152 30191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:25.751201 30192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:25.751441 30194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:25.752223 30186 server_base.cc:1034] running on GCE node
I20260926 02:24:25.752905 30186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:25.753955 30186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:25.755225 30186 hybrid_clock.cc:648] HybridClock initialized: now 1790389465755157 us; error 46 us; skew 500 ppm
I20260926 02:24:25.755642 30186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:25 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389465, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:25.759274 30186 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:24:25.760927 30186 webserver.cc:460] Webserver started at http://127.12.244.194:34253/ using document root <none> and password file <none>
I20260926 02:24:25.761711 30186 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:25.761848 30186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:25.767987 30186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:24:25.771044 30201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:25.772758 30186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:24:25.772962 30186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "ecc901c868fb4be8a7a2eeeab3b72059"
format_stamp: "Formatted at 2026-09-26 02:23:39 on dist-test-slave-qfmw"
server_key: "43ce1f0d57a4962bfb5f314bfa37b45b"
server_key_iv: "5a459a58191cadf686f64b1d6bde9af4"
server_key_version: "encryptionkey@0"
I20260926 02:24:25.774011 30186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:25.794216 30186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:25.798766 30186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:25.799075 30186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:25.800222 30186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:25.801683 30186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:25.801812 30186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:25.801985 30186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:25.802100 30186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:25.837692 30186 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:40775
I20260926 02:24:25.837786 30314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:40775 every 8 connection(s)
I20260926 02:24:25.839233 30186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:24:25.844542 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:25.842323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:52353)
0926 02:24:25.843225 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:25.843517 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":313,"threads_started":1}
W20260926 02:24:25.845268 30315 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:36095 (0 consecutive failures): Network error: Failed to ping master at 127.12.244.254:36095: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
I20260926 02:24:25.846423 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:25.846038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:47285)
0926 02:24:25.846225 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:25.846301 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 02:24:25.846434 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 30186
I20260926 02:24:25.847247 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:38483
--local_ip_for_outbound_sockets=127.12.244.195
--tserver_master_addrs=127.12.244.254:36095
--webserver_port=39281
--webserver_interface=127.12.244.195
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:24:25.847502 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:25.847123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:43251)
0926 02:24:25.847243 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:25.847317 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 02:24:25.847787 30315 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 02:24:26.203008 30318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:26.203393 30318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:26.203500 30318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:26.210981 30318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:26.211181 30318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:26.211409 30318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:24:26.221815 30318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:38483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=39281
--tserver_master_addrs=127.12.244.254:36095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:26.223245 30318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:26.224571 30318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:26.238759 30323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:26.238837 30326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:26.238827 30324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:26.239957 30318 server_base.cc:1034] running on GCE node
I20260926 02:24:26.240732 30318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:26.242024 30318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:26.243297 30318 hybrid_clock.cc:648] HybridClock initialized: now 1790389466243231 us; error 58 us; skew 500 ppm
I20260926 02:24:26.243839 30318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:26 dist-test-slave-qfmw krb5kdc[28853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389466, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:26.248598 30318 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:24:26.250344 30318 webserver.cc:460] Webserver started at http://127.12.244.195:39281/ using document root <none> and password file <none>
I20260926 02:24:26.251171 30318 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:26.251329 30318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:26.257498 30318 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 02:24:26.260740 30333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:26.262403 30318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:24:26.262660 30318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "f995a6fd37714b8f8ab427d21fa28731"
format_stamp: "Formatted at 2026-09-26 02:23:40 on dist-test-slave-qfmw"
server_key: "e83137734c93ecb4d0e5438ba63840e6"
server_key_iv: "a1413a3a17036a62fb422640b89bb879"
server_key_version: "encryptionkey@0"
I20260926 02:24:26.263967 30318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:26.294780 30318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:26.299345 30318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:26.299661 30318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:26.300699 30318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:26.302111 30318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:26.302243 30318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:26.302379 30318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:26.302467 30318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:26.339725 30318 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:38483
I20260926 02:24:26.339807 30446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:38483 every 8 connection(s)
I20260926 02:24:26.341410 30318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:24:26.344552 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 30318
I20260926 02:24:26.348384 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:26.345528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:38297)
0926 02:24:26.346825 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:26.347177 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":621,"threads_started":1}
W20260926 02:24:26.349192 30447 heartbeater.cc:644] Failed to heartbeat to 127.12.244.254:36095 (0 consecutive failures): Network error: Failed to ping master at 127.12.244.254:36095: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
I20260926 02:24:26.350236 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:26.349837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:37983)
0926 02:24:26.350024 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:26.350105 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 02:24:26.351256 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:26.350812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:47511)
0926 02:24:26.350979 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:26.351062 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:24:26.351691 30447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 02:24:26.366851 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:26.365406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:45675)
0926 02:24:26.366431 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:26.366581 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":593,"threads_started":1}
I20260926 02:24:26.849412 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:26.848354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:59857)
0926 02:24:26.849055 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:26.849179 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":398,"threads_started":1}
I20260926 02:24:27.353184 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:27.352310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:40425)
0926 02:24:27.352854 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:27.352948 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":303,"threads_started":1}
I20260926 02:24:27.369239 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:27.368193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:35591)
0926 02:24:27.368917 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:27.369018 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":403,"threads_started":1}
I20260926 02:24:27.851397 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:27.850469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:36463)
0926 02:24:27.851045 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:27.851181 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
I20260926 02:24:28.354933 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:28.354124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:36763)
0926 02:24:28.354617 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:28.354715 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":225,"threads_started":1}
I20260926 02:24:28.371016 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:28.370259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:56617)
0926 02:24:28.370727 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:28.370817 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260926 02:24:28.853241 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:28.852459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:44299)
0926 02:24:28.852944 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:28.853064 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260926 02:24:29.356812 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:29.356004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:53101)
0926 02:24:29.356548 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:29.356640 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260926 02:24:29.373030 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:29.372163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:47971)
0926 02:24:29.372701 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:29.372805 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":276,"threads_started":1}
I20260926 02:24:29.855315 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:29.854431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:53671)
0926 02:24:29.854964 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:29.855118 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260926 02:24:30.358837 30462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:30.357954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:44585)
0926 02:24:30.358519 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:30.358667 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":276,"threads_started":1}
I20260926 02:24:30.374974 30463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:30.374112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:44313)
0926 02:24:30.374701 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:30.374792 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":325,"threads_started":1}
I20260926 02:24:30.861395 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:30.857906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:51895)
0926 02:24:30.861108 (+ 3202us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:30.861211 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3046,"thread_start_us":2906,"threads_started":1}
I20260926 02:24:31.360957 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:31.359974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:56593)
0926 02:24:31.360490 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:31.360742 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":265,"threads_started":1}
I20260926 02:24:31.377010 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:31.376214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:54139)
0926 02:24:31.376738 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:31.376837 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260926 02:24:31.863431 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:31.862543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:56251)
0926 02:24:31.863042 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:31.863252 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":251,"threads_started":1}
I20260926 02:24:32.362725 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:32.361939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:42809)
0926 02:24:32.362447 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:32.362546 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":264,"threads_started":1}
I20260926 02:24:32.379395 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:32.378302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:43969)
0926 02:24:32.378977 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:32.379077 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":275,"threads_started":1}
I20260926 02:24:32.865311 30470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:32.864419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:54871)
0926 02:24:32.865000 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:32.865095 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":255,"threads_started":1}
I20260926 02:24:33.364845 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:33.363896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:43457)
0926 02:24:33.364479 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:33.364577 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":252,"threads_started":1}
I20260926 02:24:33.381314 30472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:33.380487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:38857)
0926 02:24:33.381052 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:33.381145 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":309,"threads_started":1}
I20260926 02:24:33.867408 30473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:33.866492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:55707)
0926 02:24:33.867150 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:33.867240 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":340,"threads_started":1}
I20260926 02:24:34.366762 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:34.365967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:49013)
0926 02:24:34.366487 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:34.366587 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
I20260926 02:24:34.383044 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:34.382240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:58611)
0926 02:24:34.382741 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:34.382828 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
I20260926 02:24:34.869266 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:34.868391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:42749)
0926 02:24:34.868969 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:34.869076 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":281,"threads_started":1}
I20260926 02:24:35.368906 30477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:35.367859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:40913)
0926 02:24:35.368564 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:35.368682 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":326,"threads_started":1}
I20260926 02:24:35.385044 30478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:35.384224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:57069)
0926 02:24:35.384765 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:35.384863 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":265,"threads_started":1}
I20260926 02:24:35.870980 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:35.870276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:57441)
0926 02:24:35.870735 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:35.870815 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":265,"threads_started":1}
I20260926 02:24:36.370812 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:36.370107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:54025)
0926 02:24:36.370561 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:36.370647 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":253,"threads_started":1}
I20260926 02:24:36.386972 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:36.386121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:49519)
0926 02:24:36.386691 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:36.386794 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
I20260926 02:24:36.872859 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:36.872083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:42891)
0926 02:24:36.872579 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:36.872677 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260926 02:24:37.372701 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:37.371890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:43723)
0926 02:24:37.372423 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:37.372522 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":310,"threads_started":1}
I20260926 02:24:37.388932 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:37.388123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:54145)
0926 02:24:37.388654 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:37.388754 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":295,"threads_started":1}
I20260926 02:24:37.874750 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:37.873933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:46067)
0926 02:24:37.874494 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:37.874588 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":280,"threads_started":1}
I20260926 02:24:38.374538 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:38.373787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:51789)
0926 02:24:38.374279 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:38.374367 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":274,"threads_started":1}
I20260926 02:24:38.390935 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:38.390067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:35321)
0926 02:24:38.390627 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:38.390738 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":270,"threads_started":1}
I20260926 02:24:38.876777 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:38.875900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:43329)
0926 02:24:38.876427 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:38.876586 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":248,"threads_started":1}
I20260926 02:24:39.376559 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:39.375720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:37065)
0926 02:24:39.376236 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:39.376360 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":278,"threads_started":1}
I20260926 02:24:39.392845 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:39.392011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:41861)
0926 02:24:39.392551 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:39.392645 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
I20260926 02:24:39.878892 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:39.877905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:49343)
0926 02:24:39.878475 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:39.878616 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":288,"threads_started":1}
I20260926 02:24:40.378958 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:40.377720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:53567)
0926 02:24:40.378534 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:40.378714 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":470,"threads_started":1}
I20260926 02:24:40.395224 30493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:40.394183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:33457)
0926 02:24:40.394806 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:40.394951 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":308,"threads_started":1}
I20260926 02:24:40.881119 30494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:40.880269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.194:58403)
0926 02:24:40.880828 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:40.880936 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":318,"threads_started":1}
I20260926 02:24:41.356647 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 30012
I20260926 02:24:41.375958 30495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:41.357948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.12.244.193:38281)
0926 02:24:41.358589 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:41.358607 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:41.358647 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:24:41.360365 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:41.360376 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:41.360828 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:41.361208 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:41.362786 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:41.364133 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:41.365448 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:41.366164 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:41.366386 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:41.369365 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:41.369378 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:41.369401 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:41.369451 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:41.372462 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:41.373187 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:41.373193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:41.373201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:41.373377 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:41.373782 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:41.373791 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:41.373793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:41.374353 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:41.374673 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:41.375226 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 02:24:41.375669 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":271,"threads_started":1}
I20260926 02:24:41.377559 30178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:41.381376 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:41.380467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.195:44967)
0926 02:24:41.381045 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:41.381154 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":246,"threads_started":1}
I20260926 02:24:41.465286 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:41.464140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:36095 (local address 127.12.244.193:42651)
0926 02:24:41.464700 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:41.464953 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.254:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":248,"threads_started":1}
I20260926 02:24:41.465274 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 30012
I20260926 02:24:41.483866 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 30186
I20260926 02:24:41.502319 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:41.484825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.12.244.194:40775)
0926 02:24:41.485447 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:41.485461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:41.485501 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:24:41.487076 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:41.487087 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:41.487589 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:41.487925 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:41.489256 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:41.490520 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:41.491950 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:41.492656 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:41.492791 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:41.495524 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:41.495533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:41.495550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:41.495600 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:41.498661 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:41.499566 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:41.499575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:41.499584 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:41.499774 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:41.500181 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:41.500185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:41.500187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:41.500761 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:41.501103 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:41.501575 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 02:24:41.502024 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":343,"threads_started":1}
I20260926 02:24:41.503775 30310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:41.592100 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 30186
I20260926 02:24:41.609840 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 30318
I20260926 02:24:41.630836 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:41.610899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.12.244.195:38483)
0926 02:24:41.611541 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:41.611556 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:41.611641 (+ 85us) server_negotiation.cc:407] Connection header received
0926 02:24:41.613322 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:41.613333 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:41.613794 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:41.614122 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:41.616009 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:41.617373 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:41.618832 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:41.619696 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:41.619844 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:41.622582 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:41.622590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:41.622608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:41.622652 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:41.626204 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:41.627059 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:41.627069 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:41.627080 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:41.627476 (+ 396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:41.628038 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:41.628045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:41.628052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:41.628831 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:41.629198 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:41.629762 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 02:24:41.630353 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":339,"threads_started":1}
I20260926 02:24:41.632634 30442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:41.719347 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 30318
2026-09-26T02:24:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64917 ms)
[----------] 30 tests from SecurityITest (312523 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:24:41 dist-test-slave-qfmw krb5kdc[30505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:24:41 dist-test-slave-qfmw krb5kdc[30505](info): set up 2 sockets
Sep 26 02:24:41 dist-test-slave-qfmw krb5kdc[30505](info): commencing operation
krb5kdc: starting...
W20260926 02:24:42.865813 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:24:43 dist-test-slave-qfmw krb5kdc[30505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:24:43.140509 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34441
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:34441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:43.493952 30518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:43.494347 30518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:43.494536 30518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:43.502208 30518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:24:43.502424 30518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:43.502542 30518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:43.502646 30518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:24:43.502731 30518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:24:43.513410 30518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:34441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:43.514935 30518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:43.516436 30518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:43.529762 30526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:43.529919 30524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:43.530052 30523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:43.529961 30518 server_base.cc:1034] running on GCE node
I20260926 02:24:43.531447 30518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:43.533043 30518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:43.534343 30518 hybrid_clock.cc:648] HybridClock initialized: now 1790389483534283 us; error 42 us; skew 500 ppm
I20260926 02:24:43.534760 30518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:43 dist-test-slave-qfmw krb5kdc[30505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:43.538333 30518 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:24:43.540297 30518 webserver.cc:460] Webserver started at http://127.12.244.254:43883/ using document root <none> and password file <none>
I20260926 02:24:43.541168 30518 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:43.541340 30518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:43.541726 30518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:43.545342 30518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "b255f793a9e34d4eb07c46deae0cf48e"
format_stamp: "Formatted at 2026-09-26 02:24:43 on dist-test-slave-qfmw"
server_key: "aa936ea19e3fd1940c8de0c331e8277a"
server_key_iv: "7d550f066334554446b93684997d7235"
server_key_version: "encryptionkey@0"
I20260926 02:24:43.546278 30518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "b255f793a9e34d4eb07c46deae0cf48e"
format_stamp: "Formatted at 2026-09-26 02:24:43 on dist-test-slave-qfmw"
server_key: "aa936ea19e3fd1940c8de0c331e8277a"
server_key_iv: "7d550f066334554446b93684997d7235"
server_key_version: "encryptionkey@0"
I20260926 02:24:43.552876 30518 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:24:43.557487 30533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:43.559217 30518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:24:43.559449 30518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "b255f793a9e34d4eb07c46deae0cf48e"
format_stamp: "Formatted at 2026-09-26 02:24:43 on dist-test-slave-qfmw"
server_key: "aa936ea19e3fd1940c8de0c331e8277a"
server_key_iv: "7d550f066334554446b93684997d7235"
server_key_version: "encryptionkey@0"
I20260926 02:24:43.559679 30518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:43.576720 30518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:43.581334 30518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:43.581668 30518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:43.601967 30518 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:34441
I20260926 02:24:43.601990 30584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:34441 every 8 connection(s)
I20260926 02:24:43.603530 30518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:24:43.607667 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 30518
I20260926 02:24:43.607893 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:24:43.607950 30585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:43.608786 13267 external_mini_cluster.cc:1428] Setting key 80b9448bb415fbbe26a7cae91bc20d50
I20260926 02:24:43.623371 30585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e: Bootstrap starting.
Sep 26 02:24:43 dist-test-slave-qfmw krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:43.627447 30585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:43.628671 30585 log.cc:824] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:43.631930 30585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e: No bootstrap required, opened a new log
I20260926 02:24:43.635697 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:43.611064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.12.244.254:34441)
0926 02:24:43.612014 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:43.612026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:43.612071 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:24:43.614641 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:43.614677 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:43.615222 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:43.615650 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:43.617656 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:43.619077 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:43.621135 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:43.621919 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:43.622060 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:43.627162 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:43.627178 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:43.627195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:43.627241 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:43.630655 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:43.631577 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:43.631585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:43.631592 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:43.631768 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:43.632212 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:43.632220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:43.632225 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:43.632818 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:43.633148 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:43.633668 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 02:24:43.634155 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":442,"threads_started":1}
I20260926 02:24:43.641804 30585 raft_consensus.cc:348] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255f793a9e34d4eb07c46deae0cf48e" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34441 } }
I20260926 02:24:43.642308 30585 raft_consensus.cc:374] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:43.642469 30585 raft_consensus.cc:729] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b255f793a9e34d4eb07c46deae0cf48e, State: Initialized, Role: FOLLOWER
I20260926 02:24:43.643193 30585 consensus_queue.cc:260] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255f793a9e34d4eb07c46deae0cf48e" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34441 } }
I20260926 02:24:43.643481 30585 raft_consensus.cc:388] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:24:43.643646 30585 raft_consensus.cc:482] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:24:43.643842 30585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:43.646194 30585 raft_consensus.cc:504] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255f793a9e34d4eb07c46deae0cf48e" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34441 } }
I20260926 02:24:43.646768 30585 leader_election.cc:302] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b255f793a9e34d4eb07c46deae0cf48e; no voters:
I20260926 02:24:43.647440 30585 leader_election.cc:288] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:24:43.647619 30590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:43.649119 30590 raft_consensus.cc:686] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [term 1 LEADER]: Becoming Leader. State: Replica: b255f793a9e34d4eb07c46deae0cf48e, State: Running, Role: LEADER
I20260926 02:24:43.649782 30585 sys_catalog.cc:564] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:24:43.649663 30590 consensus_queue.cc:237] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255f793a9e34d4eb07c46deae0cf48e" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34441 } }
I20260926 02:24:43.652908 30592 sys_catalog.cc:455] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b255f793a9e34d4eb07c46deae0cf48e. Latest consensus state: current_term: 1 leader_uuid: "b255f793a9e34d4eb07c46deae0cf48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255f793a9e34d4eb07c46deae0cf48e" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34441 } } }
I20260926 02:24:43.653453 30592 sys_catalog.cc:458] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:43.653970 30591 sys_catalog.cc:455] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b255f793a9e34d4eb07c46deae0cf48e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b255f793a9e34d4eb07c46deae0cf48e" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34441 } } }
I20260926 02:24:43.654259 30591 sys_catalog.cc:458] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:43.655579 30599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:24:43.660465 30599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:24:43.670040 30599 catalog_manager.cc:1347] Generated new cluster ID: f7501af07b7348b19327b9a2da9ae22b
I20260926 02:24:43.670188 30599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:24:43.681012 30599 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:24:43.682520 30599 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:24:43.689919 30599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e: Generated new TSK 0
I20260926 02:24:43.690896 30599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:24:43.756034 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34441
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:44.121006 30611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:44.121315 30611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:44.121487 30611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:44.129112 30611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:44.129248 30611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:44.129458 30611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:24:44.140338 30611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:44.141888 30611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:44.143458 30611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:44.158110 30619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:44.158227 30616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:44.158224 30617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:44.159415 30611 server_base.cc:1034] running on GCE node
I20260926 02:24:44.160154 30611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:44.161247 30611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:44.162608 30611 hybrid_clock.cc:648] HybridClock initialized: now 1790389484162486 us; error 64 us; skew 500 ppm
I20260926 02:24:44.163021 30611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:44 dist-test-slave-qfmw krb5kdc[30505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:44.166817 30611 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:24:44.168607 30611 webserver.cc:460] Webserver started at http://127.12.244.193:42405/ using document root <none> and password file <none>
I20260926 02:24:44.169392 30611 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:44.169557 30611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:44.169914 30611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:44.173346 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "98d19dbc1d12459795c350351b2c6e0a"
format_stamp: "Formatted at 2026-09-26 02:24:44 on dist-test-slave-qfmw"
server_key: "c72e787fcc64959efc3c78a14329344e"
server_key_iv: "f65268952583a2719c715662b23766b5"
server_key_version: "encryptionkey@0"
I20260926 02:24:44.174225 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "98d19dbc1d12459795c350351b2c6e0a"
format_stamp: "Formatted at 2026-09-26 02:24:44 on dist-test-slave-qfmw"
server_key: "c72e787fcc64959efc3c78a14329344e"
server_key_iv: "f65268952583a2719c715662b23766b5"
server_key_version: "encryptionkey@0"
I20260926 02:24:44.180442 30611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 02:24:44.184710 30626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:44.186403 30611 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:24:44.186627 30611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "98d19dbc1d12459795c350351b2c6e0a"
format_stamp: "Formatted at 2026-09-26 02:24:44 on dist-test-slave-qfmw"
server_key: "c72e787fcc64959efc3c78a14329344e"
server_key_iv: "f65268952583a2719c715662b23766b5"
server_key_version: "encryptionkey@0"
I20260926 02:24:44.186862 30611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:44.200860 30611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:44.205628 30611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:44.205968 30611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:44.207031 30611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:44.208638 30611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:44.208753 30611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:44.208918 30611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:44.208992 30611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:44.245980 30611 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:39209
I20260926 02:24:44.246014 30739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:39209 every 8 connection(s)
I20260926 02:24:44.247427 30611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:24:44.253785 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 30611
I20260926 02:24:44.254050 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:24:44.254806 13267 external_mini_cluster.cc:1428] Setting key ed045255e64ebfb4d616528b69031e64
Sep 26 02:24:44 dist-test-slave-qfmw krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:44.272466 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:44.250225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:42983 (local address 127.12.244.254:34441)
0926 02:24:44.250847 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:44.250852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:44.251846 (+ 994us) server_negotiation.cc:407] Connection header received
0926 02:24:44.253801 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:44.253807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:44.253949 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:44.254093 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:44.256118 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.257037 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.259647 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.260116 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.260286 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:44.264746 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:44.264753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:44.264756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:44.264797 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:44.268210 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:44.269515 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:44.269520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:44.269523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:44.269682 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:44.270339 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:44.270343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:44.270345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:44.270654 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:44.270780 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:44.271973 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0926 02:24:44.272207 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":373,"threads_started":1}
I20260926 02:24:44.274226 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:44.250827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34441 (local address 127.12.244.193:42983)
0926 02:24:44.251663 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:44.251710 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:44.253495 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:44.254398 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:44.254408 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:44.255132 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:44.255908 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:44.255936 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.257270 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.257276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:44.259450 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:44.259473 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.260416 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.260424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:44.260718 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:44.261598 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:44.261626 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:44.264434 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:44.268439 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:44.268454 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:44.268478 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:44.269307 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:44.269835 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:44.269848 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:44.269855 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:44.270194 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:44.270814 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:44.270827 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:44.271443 (+ 616us) client_negotiation.cc:769] Sending connection context
0926 02:24:44.271820 (+ 377us) client_negotiation.cc:240] Negotiation successful
0926 02:24:44.272535 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":381,"threads_started":1}
I20260926 02:24:44.276212 30740 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34441
I20260926 02:24:44.276654 30740 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:44.277542 30740 heartbeater.cc:511] Master 127.12.244.254:34441 requested a full tablet report, sending...
I20260926 02:24:44.280470 30550 ts_manager.cc:194] Registered new tserver with Master: 98d19dbc1d12459795c350351b2c6e0a (127.12.244.193:39209)
I20260926 02:24:44.282984 30550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:42983
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:24:44.319445 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34441
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:44.671602 30746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:44.671919 30746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:44.672102 30746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:44.680092 30746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:44.680224 30746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:44.680454 30746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:24:44.691226 30746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:44.692865 30746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:44.694530 30746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:44.711724 30751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:44.711759 30752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:44.711802 30754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:44.712641 30746 server_base.cc:1034] running on GCE node
I20260926 02:24:44.713444 30746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:44.714514 30746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:44.715770 30746 hybrid_clock.cc:648] HybridClock initialized: now 1790389484715693 us; error 64 us; skew 500 ppm
I20260926 02:24:44.716145 30746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:44 dist-test-slave-qfmw krb5kdc[30505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:44.719715 30746 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:24:44.721422 30746 webserver.cc:460] Webserver started at http://127.12.244.194:38515/ using document root <none> and password file <none>
I20260926 02:24:44.722179 30746 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:44.722333 30746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:44.722674 30746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:44.725847 30746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "576c31cfa0734402b43e0cc9c704d0e4"
format_stamp: "Formatted at 2026-09-26 02:24:44 on dist-test-slave-qfmw"
server_key: "04356cdb056b9cfb0e478d0bff09b1f2"
server_key_iv: "1aedefa4f7fcdde2cb0d26917a3384ff"
server_key_version: "encryptionkey@0"
I20260926 02:24:44.726687 30746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "576c31cfa0734402b43e0cc9c704d0e4"
format_stamp: "Formatted at 2026-09-26 02:24:44 on dist-test-slave-qfmw"
server_key: "04356cdb056b9cfb0e478d0bff09b1f2"
server_key_iv: "1aedefa4f7fcdde2cb0d26917a3384ff"
server_key_version: "encryptionkey@0"
I20260926 02:24:44.732587 30746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 02:24:44.736546 30761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:44.738242 30746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:24:44.738443 30746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "576c31cfa0734402b43e0cc9c704d0e4"
format_stamp: "Formatted at 2026-09-26 02:24:44 on dist-test-slave-qfmw"
server_key: "04356cdb056b9cfb0e478d0bff09b1f2"
server_key_iv: "1aedefa4f7fcdde2cb0d26917a3384ff"
server_key_version: "encryptionkey@0"
I20260926 02:24:44.738679 30746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:44.752460 30746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:44.757222 30746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:44.757539 30746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:44.758574 30746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:44.760169 30746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:44.760270 30746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:44.760424 30746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:44.760497 30746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:44.795706 30746 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:39181
I20260926 02:24:44.795753 30874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:39181 every 8 connection(s)
I20260926 02:24:44.797027 30746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:24:44.807008 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 30746
I20260926 02:24:44.807257 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:24:44.807742 13267 external_mini_cluster.cc:1428] Setting key 2e1f46f12f41b6d1246da721d5239bd8
Sep 26 02:24:44 dist-test-slave-qfmw krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:44.818991 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:44.799808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:56801 (local address 127.12.244.254:34441)
0926 02:24:44.800306 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:44.800313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:44.801402 (+ 1089us) server_negotiation.cc:407] Connection header received
0926 02:24:44.803318 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:44.803323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:44.803410 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:44.803558 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:44.805262 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.806071 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.807874 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.808341 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.808494 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:44.812535 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:44.812541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:44.812544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:44.812582 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:44.815395 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:44.816237 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:44.816241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:44.816244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:44.816385 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:44.816876 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:44.816887 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:44.816890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:44.817230 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:44.817525 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:44.818342 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 02:24:44.818583 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260926 02:24:44.820132 30878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:44.800442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34441 (local address 127.12.244.194:56801)
0926 02:24:44.801215 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:44.801257 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:44.802952 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:44.803790 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:44.803800 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:44.804401 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:44.805042 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:44.805061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.806233 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.806238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:44.807712 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:44.807728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:44.808608 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:44.808616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:44.808873 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:44.809597 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:44.809622 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:44.812255 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:44.815555 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:44.815566 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:44.815588 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:44.816079 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:44.816527 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:44.816532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:44.816536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:44.816745 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:44.817488 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:44.817497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:44.817945 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 02:24:44.818241 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 02:24:44.818852 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":286,"threads_started":1}
I20260926 02:24:44.821808 30875 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34441
I20260926 02:24:44.822176 30875 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:44.823127 30875 heartbeater.cc:511] Master 127.12.244.254:34441 requested a full tablet report, sending...
I20260926 02:24:44.824934 30550 ts_manager.cc:194] Registered new tserver with Master: 576c31cfa0734402b43e0cc9c704d0e4 (127.12.244.194:39181)
I20260926 02:24:44.826256 30550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:56801
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:24:44.867419 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34441
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:45.230552 30881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:45.230882 30881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:45.230983 30881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:45.238696 30881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:45.238862 30881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:45.239060 30881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:24:45.249782 30881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:45.251281 30881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:45.252739 30881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:45.267494 30886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:45.267518 30889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:45.267542 30887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:45.268409 30881 server_base.cc:1034] running on GCE node
I20260926 02:24:45.269129 30881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:45.270262 30881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:45.271548 30881 hybrid_clock.cc:648] HybridClock initialized: now 1790389485271474 us; error 73 us; skew 500 ppm
I20260926 02:24:45.271947 30881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:45 dist-test-slave-qfmw krb5kdc[30505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:45.275704 30881 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:24:45.277433 30881 webserver.cc:460] Webserver started at http://127.12.244.195:44713/ using document root <none> and password file <none>
I20260926 02:24:45.278215 30881 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:45.278376 30881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:45.278724 30881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:45.282073 30881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "2d7674ada0304b7f853d99ca4bfddef0"
format_stamp: "Formatted at 2026-09-26 02:24:45 on dist-test-slave-qfmw"
server_key: "286447a0c594154fd083e730f08f786f"
server_key_iv: "25aee885557a6503f4033177974ed638"
server_key_version: "encryptionkey@0"
I20260926 02:24:45.282964 30881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "2d7674ada0304b7f853d99ca4bfddef0"
format_stamp: "Formatted at 2026-09-26 02:24:45 on dist-test-slave-qfmw"
server_key: "286447a0c594154fd083e730f08f786f"
server_key_iv: "25aee885557a6503f4033177974ed638"
server_key_version: "encryptionkey@0"
I20260926 02:24:45.286907 30740 heartbeater.cc:503] Master 127.12.244.254:34441 was elected leader, sending a full tablet report...
I20260926 02:24:45.289299 30881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 02:24:45.293440 30896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:45.295083 30881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:24:45.295331 30881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "2d7674ada0304b7f853d99ca4bfddef0"
format_stamp: "Formatted at 2026-09-26 02:24:45 on dist-test-slave-qfmw"
server_key: "286447a0c594154fd083e730f08f786f"
server_key_iv: "25aee885557a6503f4033177974ed638"
server_key_version: "encryptionkey@0"
I20260926 02:24:45.295588 30881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:45.338121 30881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:45.342979 30881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:45.343338 30881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:45.344359 30881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:45.345844 30881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:45.345920 30881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:45.346012 30881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:45.346066 30881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:45.384620 30881 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:34491
I20260926 02:24:45.384730 31009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:34491 every 8 connection(s)
I20260926 02:24:45.386113 30881 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:24:45.394912 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 30881
I20260926 02:24:45.395188 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:24:45.395638 13267 external_mini_cluster.cc:1428] Setting key 024e6d8aefbe3f65faa9cd1adaa55245
Sep 26 02:24:45 dist-test-slave-qfmw krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389485, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:45.413816 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.389097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:40479 (local address 127.12.244.254:34441)
0926 02:24:45.389618 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.389626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.390695 (+ 1069us) server_negotiation.cc:407] Connection header received
0926 02:24:45.392836 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.392841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.392948 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.393055 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:45.394839 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.395883 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.397880 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.398574 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.398725 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.406762 (+ 8037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:45.406769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:45.406772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:45.406815 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:45.409737 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:45.410918 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:45.410924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:45.410927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:45.411180 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:45.411797 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:45.411801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:45.411803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:45.412243 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:45.412416 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.413328 (+ 912us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.413535 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":235,"threads_started":1}
I20260926 02:24:45.414880 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.389679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34441 (local address 127.12.244.195:40479)
0926 02:24:45.390495 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:45.390546 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:45.392515 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:45.393335 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:45.393345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:45.393971 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:45.394611 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.394630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.396079 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.396083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.397690 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.397704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.398852 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.398896 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.399295 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.403870 (+ 4575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:45.403902 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:45.406431 (+ 2529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:45.409949 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:45.409957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:45.409975 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:45.410704 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:45.411347 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:45.411354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:45.411360 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:45.411632 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:45.412420 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:45.412431 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:45.412924 (+ 493us) client_negotiation.cc:769] Sending connection context
0926 02:24:45.413221 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 02:24:45.413671 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":329,"threads_started":1}
I20260926 02:24:45.416724 31010 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34441
I20260926 02:24:45.417151 31010 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:45.417943 31010 heartbeater.cc:511] Master 127.12.244.254:34441 requested a full tablet report, sending...
I20260926 02:24:45.419703 30550 ts_manager.cc:194] Registered new tserver with Master: 2d7674ada0304b7f853d99ca4bfddef0 (127.12.244.195:34491)
I20260926 02:24:45.420526 30550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:40479
I20260926 02:24:45.425242 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:24:45.444136 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.430051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.12.244.254:34441)
0926 02:24:45.430231 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.430236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.430739 (+ 503us) server_negotiation.cc:407] Connection header received
0926 02:24:45.430893 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.430897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.430988 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.431080 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:45.432655 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.433439 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.434772 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.435329 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.435457 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.438061 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:45.438067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:45.438070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:45.438108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:45.441031 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:45.441801 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:45.441805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:45.441808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:45.441977 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:45.442505 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:45.442509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:45.442511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:45.442859 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:45.443152 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.443672 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.443863 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:24:45.448813 30550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:24:45.456724 30550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:24:45.506865 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.473223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.12.244.195:34491)
0926 02:24:45.495800 (+ 22577us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.495814 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.497858 (+ 2044us) server_negotiation.cc:407] Connection header received
0926 02:24:45.497967 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.497974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.498152 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.498384 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:45.499225 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.501468 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.504109 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.505484 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.505721 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.505900 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.506275 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.506455 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22342,"thread_start_us":22157,"threads_started":1}
I20260926 02:24:45.506904 31023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.472351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:34491 (local address 127.0.0.1:39986)
0926 02:24:45.497542 (+ 25191us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:45.497562 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:45.497705 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:45.498474 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:45.498479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:45.498505 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:45.498986 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.498999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.501624 (+ 2625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.501629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.503916 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.503930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.505720 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.505727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.506059 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.506083 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:24:45.506213 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 02:24:45.506449 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25056,"mutex_wait_us":111,"spinlock_wait_cycles":2944,"thread_start_us":24383,"threads_started":1}
I20260926 02:24:45.506896 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.471837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.12.244.194:39181)
0926 02:24:45.496745 (+ 24908us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.496753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.496786 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:24:45.496906 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.496916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.497236 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.498244 (+ 1008us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:45.498504 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.500168 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.503644 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.505060 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.505241 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.505441 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.506353 (+ 912us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.506542 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24737,"thread_start_us":24610,"threads_started":1}
I20260926 02:24:45.507432 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.472442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.12.244.193:39209)
0926 02:24:45.496393 (+ 23951us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.496404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.497225 (+ 821us) server_negotiation.cc:407] Connection header received
0926 02:24:45.497335 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.497343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.497620 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.497823 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:45.498996 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.500790 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.503438 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.505055 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.505269 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.505449 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.506857 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.507035 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23754,"thread_start_us":23584,"threads_started":1}
I20260926 02:24:45.507683 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.471594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:39181 (local address 127.0.0.1:53882)
0926 02:24:45.490914 (+ 19320us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:45.490966 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:45.491224 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:45.497632 (+ 6408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:45.497641 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:45.497713 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:45.498213 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.498225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.500304 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.500311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.503384 (+ 3073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.503401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.505654 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.505659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.505969 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.506066 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 02:24:45.506199 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 02:24:45.507257 (+ 1058us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19026,"mutex_wait_us":907,"spinlock_wait_cycles":20352,"thread_start_us":25618,"threads_started":1}
I20260926 02:24:45.508247 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.471776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39209 (local address 127.0.0.1:43912)
0926 02:24:45.496888 (+ 25112us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:45.496919 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:45.497079 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:45.498091 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:45.498095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:45.498125 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:45.498818 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.498832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.500965 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.500971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.503259 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.503274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.505629 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.505641 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.506002 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.506074 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 02:24:45.506199 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 02:24:45.507628 (+ 1429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24969,"mutex_wait_us":369,"thread_start_us":18651,"threads_started":1}
I20260926 02:24:45.511672 30810 tablet_service.cc:1467] Processing CreateTablet for tablet 66a0ea7137ff494395cba7a38fca2586 (DEFAULT_TABLE table=test-table [id=346e8cc9cc6342b28bb84af8b0532837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:45.511678 30945 tablet_service.cc:1467] Processing CreateTablet for tablet 66a0ea7137ff494395cba7a38fca2586 (DEFAULT_TABLE table=test-table [id=346e8cc9cc6342b28bb84af8b0532837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:45.512982 30810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a0ea7137ff494395cba7a38fca2586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:45.513155 30945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a0ea7137ff494395cba7a38fca2586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:45.512934 30675 tablet_service.cc:1467] Processing CreateTablet for tablet 66a0ea7137ff494395cba7a38fca2586 (DEFAULT_TABLE table=test-table [id=346e8cc9cc6342b28bb84af8b0532837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:45.514793 30675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a0ea7137ff494395cba7a38fca2586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:45.525557 31029 tablet_bootstrap.cc:492] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Bootstrap starting.
I20260926 02:24:45.529707 31029 tablet_bootstrap.cc:654] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:45.531001 31031 tablet_bootstrap.cc:492] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Bootstrap starting.
I20260926 02:24:45.531144 31029 log.cc:824] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:45.533087 31030 tablet_bootstrap.cc:492] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Bootstrap starting.
I20260926 02:24:45.536217 31031 tablet_bootstrap.cc:654] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:45.536414 31030 tablet_bootstrap.cc:654] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:45.537767 31030 log.cc:824] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:45.538323 31029 tablet_bootstrap.cc:492] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: No bootstrap required, opened a new log
I20260926 02:24:45.538431 31031 log.cc:824] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:45.538648 31029 ts_tablet_manager.cc:1397] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260926 02:24:45.541184 31031 tablet_bootstrap.cc:492] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: No bootstrap required, opened a new log
I20260926 02:24:45.541333 31030 tablet_bootstrap.cc:492] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: No bootstrap required, opened a new log
I20260926 02:24:45.541541 31031 ts_tablet_manager.cc:1397] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260926 02:24:45.541623 31030 ts_tablet_manager.cc:1397] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 02:24:45.549853 31031 raft_consensus.cc:348] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.550335 31031 raft_consensus.cc:374] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:45.550449 31031 raft_consensus.cc:729] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d19dbc1d12459795c350351b2c6e0a, State: Initialized, Role: FOLLOWER
I20260926 02:24:45.550266 31029 raft_consensus.cc:348] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.550765 31029 raft_consensus.cc:374] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:45.550870 31029 raft_consensus.cc:729] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576c31cfa0734402b43e0cc9c704d0e4, State: Initialized, Role: FOLLOWER
I20260926 02:24:45.551247 31031 consensus_queue.cc:260] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.551615 31029 consensus_queue.cc:260] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.553416 31031 ts_tablet_manager.cc:1428] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 02:24:45.553170 31030 raft_consensus.cc:348] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.553772 31030 raft_consensus.cc:374] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:45.553884 31030 raft_consensus.cc:729] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7674ada0304b7f853d99ca4bfddef0, State: Initialized, Role: FOLLOWER
I20260926 02:24:45.554888 31030 consensus_queue.cc:260] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.557121 31010 heartbeater.cc:503] Master 127.12.244.254:34441 was elected leader, sending a full tablet report...
I20260926 02:24:45.557289 31030 ts_tablet_manager.cc:1428] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260926 02:24:45.558853 30875 heartbeater.cc:503] Master 127.12.244.254:34441 was elected leader, sending a full tablet report...
I20260926 02:24:45.560139 31029 ts_tablet_manager.cc:1428] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Time spent starting tablet: real 0.021s user 0.016s sys 0.006s
W20260926 02:24:45.638060 31011 tablet.cc:2369] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:45.750159 30741 tablet.cc:2369] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:45.799472 30876 tablet.cc:2369] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:24:45.870266 31035 raft_consensus.cc:482] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:24:45.870704 31035 raft_consensus.cc:504] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.872800 31035 leader_election.cc:288] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 576c31cfa0734402b43e0cc9c704d0e4 (127.12.244.194:39181), 2d7674ada0304b7f853d99ca4bfddef0 (127.12.244.195:34491)
I20260926 02:24:45.880686 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.873288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:33081 (local address 127.12.244.194:39181)
0926 02:24:45.873539 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.873545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.874493 (+ 948us) server_negotiation.cc:407] Connection header received
0926 02:24:45.874669 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.874673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.874736 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.874803 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:45.875648 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.876389 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.878587 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.879494 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.879709 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.879805 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.880224 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.880367 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:24:45.880688 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.873342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:48001 (local address 127.12.244.195:34491)
0926 02:24:45.873598 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.873609 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.874395 (+ 786us) server_negotiation.cc:407] Connection header received
0926 02:24:45.874692 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.874697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.874761 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.874895 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:45.875683 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.876406 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.878475 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.879489 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.879710 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.879804 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.880219 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.880372 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:24:45.880913 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.873221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:39181 (local address 127.12.244.193:33081)
0926 02:24:45.874380 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:45.874396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:45.874521 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:45.874858 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:45.874864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:45.874893 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:45.875444 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.875460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.876537 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.876543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.878403 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.878422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.879759 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.879767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.880026 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.880063 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:24:45.880182 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 02:24:45.880378 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":105,"thread_start_us":726,"threads_started":1}
I20260926 02:24:45.881294 31038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.873161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:34491 (local address 127.12.244.193:48001)
0926 02:24:45.874267 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:45.874285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:45.874494 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:45.874919 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:45.874924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:45.874950 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:45.875491 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.875501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.876587 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.876590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.878310 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:45.878324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.879806 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.879813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:45.880023 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.880067 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 02:24:45.880183 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 02:24:45.880368 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":135,"thread_start_us":702,"threads_started":1}
I20260926 02:24:45.882110 30830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0ea7137ff494395cba7a38fca2586" candidate_uuid: "98d19dbc1d12459795c350351b2c6e0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c31cfa0734402b43e0cc9c704d0e4" is_pre_election: true
I20260926 02:24:45.882424 30965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0ea7137ff494395cba7a38fca2586" candidate_uuid: "98d19dbc1d12459795c350351b2c6e0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7674ada0304b7f853d99ca4bfddef0" is_pre_election: true
I20260926 02:24:45.882777 30830 raft_consensus.cc:2454] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98d19dbc1d12459795c350351b2c6e0a in term 0.
I20260926 02:24:45.882901 30965 raft_consensus.cc:2454] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98d19dbc1d12459795c350351b2c6e0a in term 0.
I20260926 02:24:45.883671 30627 leader_election.cc:302] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d7674ada0304b7f853d99ca4bfddef0, 576c31cfa0734402b43e0cc9c704d0e4, 98d19dbc1d12459795c350351b2c6e0a; no voters:
I20260926 02:24:45.884287 31035 raft_consensus.cc:2781] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:24:45.884528 31035 raft_consensus.cc:482] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:24:45.884683 31035 raft_consensus.cc:3037] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:45.887207 31035 raft_consensus.cc:504] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.888056 31035 leader_election.cc:288] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [CANDIDATE]: Term 1 election: Requested vote from peers 576c31cfa0734402b43e0cc9c704d0e4 (127.12.244.194:39181), 2d7674ada0304b7f853d99ca4bfddef0 (127.12.244.195:34491)
I20260926 02:24:45.888723 30830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0ea7137ff494395cba7a38fca2586" candidate_uuid: "98d19dbc1d12459795c350351b2c6e0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c31cfa0734402b43e0cc9c704d0e4"
I20260926 02:24:45.888749 30965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a0ea7137ff494395cba7a38fca2586" candidate_uuid: "98d19dbc1d12459795c350351b2c6e0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7674ada0304b7f853d99ca4bfddef0"
I20260926 02:24:45.889046 30965 raft_consensus.cc:3037] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:45.888996 30830 raft_consensus.cc:3037] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:45.891819 30830 raft_consensus.cc:2454] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d19dbc1d12459795c350351b2c6e0a in term 1.
I20260926 02:24:45.891829 30965 raft_consensus.cc:2454] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98d19dbc1d12459795c350351b2c6e0a in term 1.
I20260926 02:24:45.892457 30629 leader_election.cc:302] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d7674ada0304b7f853d99ca4bfddef0, 576c31cfa0734402b43e0cc9c704d0e4, 98d19dbc1d12459795c350351b2c6e0a; no voters:
I20260926 02:24:45.892871 31035 raft_consensus.cc:2781] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:45.893342 31035 raft_consensus.cc:686] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 1 LEADER]: Becoming Leader. State: Replica: 98d19dbc1d12459795c350351b2c6e0a, State: Running, Role: LEADER
I20260926 02:24:45.894017 31035 consensus_queue.cc:237] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } }
I20260926 02:24:45.898901 30548 catalog_manager.cc:5495] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 98d19dbc1d12459795c350351b2c6e0a (127.12.244.193). New cstate: current_term: 1 leader_uuid: "98d19dbc1d12459795c350351b2c6e0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98d19dbc1d12459795c350351b2c6e0a" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 } health_report { overall_health: UNKNOWN } } }
I20260926 02:24:45.969934 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:45.963350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.12.244.193:39209)
0926 02:24:45.963565 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:45.963570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:45.963992 (+ 422us) server_negotiation.cc:407] Connection header received
0926 02:24:45.964213 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:45.964218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:45.964324 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:45.964466 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:24:45.965019 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.965867 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.967530 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:45.968022 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:45.968155 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:45.968690 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:24:45.969278 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:24:45.969412 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:45.969632 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 02:24:45.969715 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:24:45.979565 30830 raft_consensus.cc:1261] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 1 FOLLOWER]: Refusing update from remote peer 98d19dbc1d12459795c350351b2c6e0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:45.979565 30965 raft_consensus.cc:1261] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 1 FOLLOWER]: Refusing update from remote peer 98d19dbc1d12459795c350351b2c6e0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:45.980777 31035 consensus_queue.cc:1035] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7674ada0304b7f853d99ca4bfddef0" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 34491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:45.981124 31040 consensus_queue.cc:1035] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "576c31cfa0734402b43e0cc9c704d0e4" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:46.010492 30548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:46.010840 30548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:46.015166 30548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b255f793a9e34d4eb07c46deae0cf48e: Sending DeleteTablet for 3 replicas of tablet 66a0ea7137ff494395cba7a38fca2586
I20260926 02:24:46.016649 30810 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a0ea7137ff494395cba7a38fca2586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:46 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:53882
I20260926 02:24:46.016644 30945 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a0ea7137ff494395cba7a38fca2586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:46 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:39986
I20260926 02:24:46.016669 30675 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a0ea7137ff494395cba7a38fca2586 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:46 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:43912
I20260926 02:24:46.019408 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 30611
I20260926 02:24:46.030635 31055 tablet_replica.cc:330] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: stopping tablet replica
I20260926 02:24:46.031250 31055 raft_consensus.cc:2229] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:46.031865 31055 raft_consensus.cc:2258] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:46.032757 31056 tablet_replica.cc:330] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: stopping tablet replica
I20260926 02:24:46.033244 31056 raft_consensus.cc:2229] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:24:46.034005 31056 raft_consensus.cc:2258] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:46.036827 31056 ts_tablet_manager.cc:1905] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:46.042330 31054 tablet_replica.cc:330] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: stopping tablet replica
I20260926 02:24:46.042842 31054 raft_consensus.cc:2229] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:46.043498 31054 raft_consensus.cc:2258] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:46.044916 31055 ts_tablet_manager.cc:1905] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:46.045394 31054 ts_tablet_manager.cc:1905] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:24:46 dist-test-slave-qfmw krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:24:46.053139 31055 ts_tablet_manager.cc:1918] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:46.053354 31055 log.cc:1196] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/66a0ea7137ff494395cba7a38fca2586
I20260926 02:24:46.053947 31055 ts_tablet_manager.cc:1939] T 66a0ea7137ff494395cba7a38fca2586 P 2d7674ada0304b7f853d99ca4bfddef0: Deleting consensus metadata
I20260926 02:24:46.055752 31056 ts_tablet_manager.cc:1918] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:46.056262 30536 catalog_manager.cc:4841] TS 2d7674ada0304b7f853d99ca4bfddef0 (127.12.244.195:34491): tablet 66a0ea7137ff494395cba7a38fca2586 (table test-table [id=346e8cc9cc6342b28bb84af8b0532837]) successfully deleted
I20260926 02:24:46.056618 31056 log.cc:1196] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/66a0ea7137ff494395cba7a38fca2586
I20260926 02:24:46.057192 31056 ts_tablet_manager.cc:1939] T 66a0ea7137ff494395cba7a38fca2586 P 98d19dbc1d12459795c350351b2c6e0a: Deleting consensus metadata
I20260926 02:24:46.058125 31054 ts_tablet_manager.cc:1918] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:46.058341 31054 log.cc:1196] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/66a0ea7137ff494395cba7a38fca2586
I20260926 02:24:46.058367 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:46.021115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.12.244.193:39209)
0926 02:24:46.021377 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:46.021386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:46.031071 (+ 9685us) server_negotiation.cc:407] Connection header received
0926 02:24:46.031533 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:46.031543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:46.031702 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:46.031803 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:46.034613 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:46.036351 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:46.039183 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:46.040253 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:46.040395 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:46.047940 (+ 7545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:46.047963 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:46.047994 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:46.048056 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:46.052750 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:46.054580 (+ 1830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:46.054594 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:46.054597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:46.054818 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:46.055400 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:46.055405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:46.055407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:46.056098 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:46.056312 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:46.056948 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 02:24:46.057166 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 02:24:46.059166 30735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:46.059341 31054 ts_tablet_manager.cc:1939] T 66a0ea7137ff494395cba7a38fca2586 P 576c31cfa0734402b43e0cc9c704d0e4: Deleting consensus metadata
I20260926 02:24:46.062032 30534 catalog_manager.cc:4841] TS 576c31cfa0734402b43e0cc9c704d0e4 (127.12.244.194:39181): tablet 66a0ea7137ff494395cba7a38fca2586 (table test-table [id=346e8cc9cc6342b28bb84af8b0532837]) successfully deleted
I20260926 02:24:46.162119 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 30611
I20260926 02:24:46.164026 30537 catalog_manager.cc:4841] TS 98d19dbc1d12459795c350351b2c6e0a (127.12.244.193:39209): tablet 66a0ea7137ff494395cba7a38fca2586 (table test-table [id=346e8cc9cc6342b28bb84af8b0532837]) successfully deleted
I20260926 02:24:46.181262 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 30746
Sep 26 02:24:46 dist-test-slave-qfmw krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:24:46.197933 31025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:46.182126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.12.244.194:39181)
0926 02:24:46.182324 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:46.182333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:46.182356 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:24:46.182573 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:46.182579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:46.182673 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:46.182786 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:46.184238 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:46.184965 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:46.186328 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:46.186763 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:46.186910 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:46.191832 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:46.191850 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:46.191868 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:46.191914 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:46.195074 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:46.196014 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:46.196022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:46.196024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:46.196171 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:46.196525 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:46.196530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:46.196533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:46.196932 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:46.197177 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:46.197488 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 02:24:46.197675 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:24:46.198632 30870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:46.291697 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 30746
I20260926 02:24:46.309082 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 30881
Sep 26 02:24:46 dist-test-slave-qfmw krb5kdc[30505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:24:46.325268 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:46.310004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.12.244.195:34491)
0926 02:24:46.310182 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:46.310187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:46.310206 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:24:46.310289 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:46.310293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:46.310371 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:46.310456 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:46.312036 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:46.312819 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:46.314154 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:46.314544 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:46.314642 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:46.319403 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:46.319420 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:46.319436 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:46.319479 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:46.322489 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:46.323399 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:46.323406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:46.323409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:46.323547 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:46.323898 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:46.323901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:46.323903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:46.324280 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:46.324490 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:46.324796 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 02:24:46.324967 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:24:46.325958 31005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:46.416062 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 30881
I20260926 02:24:46.433279 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 30518
I20260926 02:24:46.434554 30580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:46.546820 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 30518
2026-09-26T02:24:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4835 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T02:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:24:46Z Disabled control of system clock
I20260926 02:24:46.630576 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33279
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:24:46.986315 31066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:46.986629 31066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:46.986717 31066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:46.994390 31066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:24:46.994519 31066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:46.994556 31066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:46.994614 31066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:24:46.994660 31066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:24:47.004498 31066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:33279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:47.005834 31066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:47.007732 31066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:47.020622 31071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:47.020738 31072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:47.020699 31074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:47.021651 31066 server_base.cc:1034] running on GCE node
I20260926 02:24:47.022594 31066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:47.024330 31066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:47.025597 31066 hybrid_clock.cc:648] HybridClock initialized: now 1790389487025519 us; error 76 us; skew 500 ppm
I20260926 02:24:47.026005 31066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:47.028728 31066 webserver.cc:460] Webserver started at http://127.12.244.254:38601/ using document root <none> and password file <none>
I20260926 02:24:47.029501 31066 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:47.029645 31066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:47.029989 31066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:47.033186 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "d663432466074ca4b97a1b46d48360df"
format_stamp: "Formatted at 2026-09-26 02:24:47 on dist-test-slave-qfmw"
server_key: "76cbcb8f6cbf98876510b601b94c1e63"
server_key_iv: "9d0498cc5e9b3349e6436cccddcf241a"
server_key_version: "encryptionkey@0"
I20260926 02:24:47.034089 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "d663432466074ca4b97a1b46d48360df"
format_stamp: "Formatted at 2026-09-26 02:24:47 on dist-test-slave-qfmw"
server_key: "76cbcb8f6cbf98876510b601b94c1e63"
server_key_iv: "9d0498cc5e9b3349e6436cccddcf241a"
server_key_version: "encryptionkey@0"
I20260926 02:24:47.039923 31066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:24:47.044059 31080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:47.045790 31066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:24:47.046032 31066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "d663432466074ca4b97a1b46d48360df"
format_stamp: "Formatted at 2026-09-26 02:24:47 on dist-test-slave-qfmw"
server_key: "76cbcb8f6cbf98876510b601b94c1e63"
server_key_iv: "9d0498cc5e9b3349e6436cccddcf241a"
server_key_version: "encryptionkey@0"
I20260926 02:24:47.046500 31066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:47.061272 31066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:47.062258 31066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:47.062597 31066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:47.083257 31066 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:33279
I20260926 02:24:47.083279 31132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:33279 every 8 connection(s)
I20260926 02:24:47.084901 31066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:24:47.087272 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31066
I20260926 02:24:47.087518 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:24:47.087978 13267 external_mini_cluster.cc:1428] Setting key 5ce1e1a54695b2ad4f3a9c2b93663449
I20260926 02:24:47.090943 31133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:47.102367 31136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:47.089805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.12.244.254:33279)
0926 02:24:47.090739 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:47.090750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:47.090793 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:24:47.092621 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:47.092644 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:47.093144 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:47.093590 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:47.094236 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:47.095697 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:47.097140 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:47.098317 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:47.098519 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:47.099838 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:47.099847 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:47.099865 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:47.100487 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:47.100567 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:47.100595 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:47.100745 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:47.101002 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 02:24:47.101357 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":298,"threads_started":1}
I20260926 02:24:47.103333 31133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df: Bootstrap starting.
I20260926 02:24:47.106871 31133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:47.108057 31133 log.cc:824] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:47.111572 31133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df: No bootstrap required, opened a new log
I20260926 02:24:47.120122 31133 raft_consensus.cc:348] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663432466074ca4b97a1b46d48360df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33279 } }
I20260926 02:24:47.120632 31133 raft_consensus.cc:374] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:47.120752 31133 raft_consensus.cc:729] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d663432466074ca4b97a1b46d48360df, State: Initialized, Role: FOLLOWER
I20260926 02:24:47.121436 31133 consensus_queue.cc:260] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663432466074ca4b97a1b46d48360df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33279 } }
I20260926 02:24:47.121687 31133 raft_consensus.cc:388] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:24:47.121842 31133 raft_consensus.cc:482] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:24:47.122040 31133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:47.124492 31133 raft_consensus.cc:504] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663432466074ca4b97a1b46d48360df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33279 } }
I20260926 02:24:47.125114 31133 leader_election.cc:302] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d663432466074ca4b97a1b46d48360df; no voters:
I20260926 02:24:47.125855 31133 leader_election.cc:288] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:24:47.125931 31138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:47.126706 31138 raft_consensus.cc:686] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [term 1 LEADER]: Becoming Leader. State: Replica: d663432466074ca4b97a1b46d48360df, State: Running, Role: LEADER
I20260926 02:24:47.127327 31138 consensus_queue.cc:237] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663432466074ca4b97a1b46d48360df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33279 } }
I20260926 02:24:47.128202 31133 sys_catalog.cc:564] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:24:47.130326 31140 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [sys.catalog]: SysCatalogTable state changed. Reason: New leader d663432466074ca4b97a1b46d48360df. Latest consensus state: current_term: 1 leader_uuid: "d663432466074ca4b97a1b46d48360df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663432466074ca4b97a1b46d48360df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33279 } } }
I20260926 02:24:47.130326 31139 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d663432466074ca4b97a1b46d48360df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663432466074ca4b97a1b46d48360df" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 33279 } } }
I20260926 02:24:47.130861 31140 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:47.130859 31139 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:47.136605 31144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:24:47.143040 31144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:24:47.155019 31144 catalog_manager.cc:1347] Generated new cluster ID: 758725fa329d452a822a57c29e59ebfa
I20260926 02:24:47.155218 31144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:24:47.168164 31144 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:24:47.169687 31144 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:24:47.181963 31144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df: Generated new TSK 0
I20260926 02:24:47.183197 31144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:24:47.197353 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33279
--builtin_ntp_servers=127.12.244.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:24:47.556391 31157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:47.556742 31157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:47.556900 31157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:47.564580 31157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:47.564733 31157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:47.564916 31157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:24:47.574774 31157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:47.576155 31157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:47.577975 31157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:47.591719 31162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:47.591657 31163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:47.591775 31165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:47.592350 31157 server_base.cc:1034] running on GCE node
I20260926 02:24:47.593031 31157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:47.594102 31157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:47.595351 31157 hybrid_clock.cc:648] HybridClock initialized: now 1790389487595293 us; error 44 us; skew 500 ppm
I20260926 02:24:47.595815 31157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:47.598667 31157 webserver.cc:460] Webserver started at http://127.12.244.193:45005/ using document root <none> and password file <none>
I20260926 02:24:47.599574 31157 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:47.599740 31157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:47.600111 31157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:47.603777 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "76272099209444af80c318661cf7dbc8"
format_stamp: "Formatted at 2026-09-26 02:24:47 on dist-test-slave-qfmw"
server_key: "e583941d31ff36e330f8cdfedd93e00a"
server_key_iv: "88f68a50e244965baa42ac9b3517b2b0"
server_key_version: "encryptionkey@0"
I20260926 02:24:47.604792 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "76272099209444af80c318661cf7dbc8"
format_stamp: "Formatted at 2026-09-26 02:24:47 on dist-test-slave-qfmw"
server_key: "e583941d31ff36e330f8cdfedd93e00a"
server_key_iv: "88f68a50e244965baa42ac9b3517b2b0"
server_key_version: "encryptionkey@0"
I20260926 02:24:47.610603 31157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:24:47.614579 31171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:47.616233 31157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:24:47.616463 31157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "76272099209444af80c318661cf7dbc8"
format_stamp: "Formatted at 2026-09-26 02:24:47 on dist-test-slave-qfmw"
server_key: "e583941d31ff36e330f8cdfedd93e00a"
server_key_iv: "88f68a50e244965baa42ac9b3517b2b0"
server_key_version: "encryptionkey@0"
I20260926 02:24:47.616711 31157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:47.644948 31157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:47.645936 31157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:47.646265 31157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:47.647390 31157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:47.648923 31157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:47.649049 31157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:47.649201 31157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:47.649289 31157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:47.685668 31157 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:36699
I20260926 02:24:47.685760 31283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:36699 every 8 connection(s)
I20260926 02:24:47.687077 31157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:24:47.696671 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31157
I20260926 02:24:47.696867 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:24:47.697463 13267 external_mini_cluster.cc:1428] Setting key cfa9be371bd51cc91ad2e7d4f7b9ca20
I20260926 02:24:47.700997 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33279
--builtin_ntp_servers=127.12.244.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:24:47.702185 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:47.690068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:55971 (local address 127.12.244.254:33279)
0926 02:24:47.690573 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:47.690578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:47.691972 (+ 1394us) server_negotiation.cc:407] Connection header received
0926 02:24:47.693838 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:47.693863 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:47.693994 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:47.694149 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:47.695474 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:47.696192 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:47.698698 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:47.699308 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:47.699449 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:47.701366 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:47.701371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:47.701374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:47.701395 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:47.701436 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:47.701448 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:47.701523 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:47.701819 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 02:24:47.701907 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":263,"threads_started":1}
I20260926 02:24:47.703943 31287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:47.690746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33279 (local address 127.12.244.193:55971)
0926 02:24:47.691801 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:47.691845 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:47.693565 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:47.694418 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:47.694428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:47.694532 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:47.695281 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:47.695309 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:47.696386 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:47.696392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:47.698526 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:47.698545 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:47.699579 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:47.699588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:47.699927 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:47.700805 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:24:47.700811 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:47.700885 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:24:47.701040 (+ 155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:47.701561 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:47.701576 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:47.701588 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:24:47.701812 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 02:24:47.702329 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260926 02:24:47.705705 31284 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33279
I20260926 02:24:47.706030 31284 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:47.706844 31284 heartbeater.cc:511] Master 127.12.244.254:33279 requested a full tablet report, sending...
I20260926 02:24:47.709299 31098 ts_manager.cc:194] Registered new tserver with Master: 76272099209444af80c318661cf7dbc8 (127.12.244.193:36699)
I20260926 02:24:47.711578 31098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.193:55971
W20260926 02:24:48.062074 31288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:48.062387 31288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:48.062477 31288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:48.070163 31288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:48.070284 31288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:48.070395 31288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:24:48.079906 31288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:48.081228 31288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:48.083194 31288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:48.097147 31294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:48.097188 31296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:48.097883 31293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:48.098418 31288 server_base.cc:1034] running on GCE node
I20260926 02:24:48.099220 31288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:48.100452 31288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:48.101732 31288 hybrid_clock.cc:648] HybridClock initialized: now 1790389488101639 us; error 82 us; skew 500 ppm
I20260926 02:24:48.102048 31288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:48.104435 31288 webserver.cc:460] Webserver started at http://127.12.244.194:41501/ using document root <none> and password file <none>
I20260926 02:24:48.105113 31288 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:48.105204 31288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:48.105499 31288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:48.108459 31288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "53ac9812551f4d5db5433c12c96f8cb1"
format_stamp: "Formatted at 2026-09-26 02:24:48 on dist-test-slave-qfmw"
server_key: "9adacef58018fd0ffb5aa6883947f951"
server_key_iv: "a0aa7c620a24cd22fd32995fc1cd8f10"
server_key_version: "encryptionkey@0"
I20260926 02:24:48.109218 31288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "53ac9812551f4d5db5433c12c96f8cb1"
format_stamp: "Formatted at 2026-09-26 02:24:48 on dist-test-slave-qfmw"
server_key: "9adacef58018fd0ffb5aa6883947f951"
server_key_iv: "a0aa7c620a24cd22fd32995fc1cd8f10"
server_key_version: "encryptionkey@0"
I20260926 02:24:48.114606 31288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:24:48.118251 31302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:48.119911 31288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:24:48.120148 31288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "53ac9812551f4d5db5433c12c96f8cb1"
format_stamp: "Formatted at 2026-09-26 02:24:48 on dist-test-slave-qfmw"
server_key: "9adacef58018fd0ffb5aa6883947f951"
server_key_iv: "a0aa7c620a24cd22fd32995fc1cd8f10"
server_key_version: "encryptionkey@0"
I20260926 02:24:48.120357 31288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:48.141498 31288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:48.142441 31288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:48.142783 31288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:48.143883 31288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:48.145496 31288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:48.145586 31288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:48.145746 31288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:48.145831 31288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:48.180423 31288 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:41321
I20260926 02:24:48.180569 31414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:41321 every 8 connection(s)
I20260926 02:24:48.181851 31288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:24:48.191341 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31288
I20260926 02:24:48.191540 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:24:48.191979 13267 external_mini_cluster.cc:1428] Setting key b0f0e4dfaa32d725d1708ca2136dd37b
I20260926 02:24:48.195183 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33279
--builtin_ntp_servers=127.12.244.212:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:24:48.195437 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.184574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:36877 (local address 127.12.244.254:33279)
0926 02:24:48.184801 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:48.184808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:48.186324 (+ 1516us) server_negotiation.cc:407] Connection header received
0926 02:24:48.188200 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:48.188205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:48.188278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:48.188370 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:48.189511 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.190294 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.192162 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.192660 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.192812 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.194559 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:48.194564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:48.194567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:48.194584 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:48.194619 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:48.194632 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:48.194731 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:48.194974 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 02:24:48.195077 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:24:48.196578 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.185119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33279 (local address 127.12.244.194:36877)
0926 02:24:48.186079 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:48.186116 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:48.187894 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:48.188588 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:48.188598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:48.188694 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:48.189314 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.189331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.190480 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.190484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.191990 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.192003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.192956 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.192964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.193280 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.194138 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:24:48.194141 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:48.194202 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:24:48.194318 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:48.194732 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:48.194746 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:48.194757 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:24:48.194963 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 02:24:48.195429 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":307,"threads_started":1}
I20260926 02:24:48.198107 31415 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33279
I20260926 02:24:48.198427 31415 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:48.199452 31415 heartbeater.cc:511] Master 127.12.244.254:33279 requested a full tablet report, sending...
I20260926 02:24:48.201049 31098 ts_manager.cc:194] Registered new tserver with Master: 53ac9812551f4d5db5433c12c96f8cb1 (127.12.244.194:41321)
I20260926 02:24:48.202301 31098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.194:36877
W20260926 02:24:48.534592 31418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:48.534901 31418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:48.535049 31418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:48.542600 31418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:48.542714 31418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:48.542822 31418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:24:48.552258 31418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:48.553567 31418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:48.555373 31418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:48.569519 31426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:48.569453 31423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:48.569618 31424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:48.570837 31418 server_base.cc:1034] running on GCE node
I20260926 02:24:48.571652 31418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:48.572726 31418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:48.573966 31418 hybrid_clock.cc:648] HybridClock initialized: now 1790389488573906 us; error 50 us; skew 500 ppm
I20260926 02:24:48.574365 31418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:48.576908 31418 webserver.cc:460] Webserver started at http://127.12.244.195:46807/ using document root <none> and password file <none>
I20260926 02:24:48.577665 31418 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:48.577782 31418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:48.578125 31418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:48.581202 31418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "8acfa083a0fa425abe37cb4544bf9b35"
format_stamp: "Formatted at 2026-09-26 02:24:48 on dist-test-slave-qfmw"
server_key: "6ed752229bcc96b3d13b791ffb7831be"
server_key_iv: "7d0f15f3300b939a66349b7b9fb60de9"
server_key_version: "encryptionkey@0"
I20260926 02:24:48.582038 31418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "8acfa083a0fa425abe37cb4544bf9b35"
format_stamp: "Formatted at 2026-09-26 02:24:48 on dist-test-slave-qfmw"
server_key: "6ed752229bcc96b3d13b791ffb7831be"
server_key_iv: "7d0f15f3300b939a66349b7b9fb60de9"
server_key_version: "encryptionkey@0"
I20260926 02:24:48.587544 31418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:24:48.591199 31432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:48.592674 31418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:24:48.592873 31418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "8acfa083a0fa425abe37cb4544bf9b35"
format_stamp: "Formatted at 2026-09-26 02:24:48 on dist-test-slave-qfmw"
server_key: "6ed752229bcc96b3d13b791ffb7831be"
server_key_iv: "7d0f15f3300b939a66349b7b9fb60de9"
server_key_version: "encryptionkey@0"
I20260926 02:24:48.593106 31418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:48.606683 31418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:48.607642 31418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:48.607973 31418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:48.608984 31418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:48.610566 31418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:48.610692 31418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:48.610841 31418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:48.610936 31418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:48.645630 31418 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:35977
I20260926 02:24:48.645718 31545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:35977 every 8 connection(s)
I20260926 02:24:48.646978 31418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:24:48.651608 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31418
I20260926 02:24:48.651813 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:24:48.652292 13267 external_mini_cluster.cc:1428] Setting key 44fd7808b1e6bc99fb115335d1521b94
I20260926 02:24:48.661043 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.649530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:55127 (local address 127.12.244.254:33279)
0926 02:24:48.649721 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:48.649726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:48.651236 (+ 1510us) server_negotiation.cc:407] Connection header received
0926 02:24:48.653380 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:48.653385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:48.653487 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:48.653598 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:48.655435 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.656572 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.658457 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.658885 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.658983 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.660355 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:48.660360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:48.660363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:48.660382 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:48.660414 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:48.660426 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:48.660495 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:48.660697 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 02:24:48.660804 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:24:48.662266 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.650073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:33279 (local address 127.12.244.195:55127)
0926 02:24:48.651004 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:48.651073 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:48.653085 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:48.654103 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:48.654138 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:48.654266 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:48.655231 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.655259 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.656744 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.656750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.658324 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.658337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.659151 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.659160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.659371 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.660058 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:24:48.660061 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:48.660102 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:24:48.660182 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:48.660510 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:48.660517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:48.660523 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:24:48.660661 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 02:24:48.660980 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":302,"threads_started":1}
I20260926 02:24:48.663864 31546 heartbeater.cc:347] Connected to a master server at 127.12.244.254:33279
I20260926 02:24:48.664163 31546 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:48.664865 31546 heartbeater.cc:511] Master 127.12.244.254:33279 requested a full tablet report, sending...
I20260926 02:24:48.666450 31098 ts_manager.cc:194] Registered new tserver with Master: 8acfa083a0fa425abe37cb4544bf9b35 (127.12.244.195:35977)
I20260926 02:24:48.667506 31098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.195:55127
I20260926 02:24:48.667858 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:24:48.678300 31286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.672298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.12.244.254:33279)
0926 02:24:48.672527 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:48.672534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:48.672727 (+ 193us) server_negotiation.cc:407] Connection header received
0926 02:24:48.672887 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:48.672891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:48.672968 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:48.673050 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:48.673660 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.674376 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.675927 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.676364 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.676477 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.677615 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:48.677620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:48.677623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:48.677642 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:48.677673 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:48.677685 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:48.677758 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:48.677881 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 02:24:48.677958 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:24:48.682324 31098 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:24:48.689399 31098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:24:48.716475 31284 heartbeater.cc:503] Master 127.12.244.254:33279 was elected leader, sending a full tablet report...
I20260926 02:24:48.736454 31560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.704655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:35977 (local address 127.0.0.1:40838)
0926 02:24:48.727498 (+ 22843us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:48.727517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:48.727654 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:48.728829 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:48.728833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:48.728858 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:48.729371 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.729385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.731074 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.731078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.733330 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.733350 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.735390 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.735400 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.735641 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.735709 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 02:24:48.735844 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 02:24:48.736060 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22704,"mutex_wait_us":18,"thread_start_us":20410,"threads_started":1}
I20260926 02:24:48.736476 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.705254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.12.244.195:35977)
0926 02:24:48.726435 (+ 21181us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:48.726446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:48.728145 (+ 1699us) server_negotiation.cc:407] Connection header received
0926 02:24:48.728266 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:48.728274 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:48.728605 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:48.728810 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:48.729541 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.730897 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.733604 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.735160 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.735382 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.735655 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:48.735913 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 02:24:48.736094 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20978,"thread_start_us":21193,"threads_started":1}
I20260926 02:24:48.737324 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.704211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51712)
0926 02:24:48.726213 (+ 22002us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:48.726276 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:48.726509 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:48.728549 (+ 2040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:48.728557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:48.728608 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:48.729166 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.729186 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.731338 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.731346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.733551 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.733562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.735582 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.735588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.735738 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.735762 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:24:48.735818 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 02:24:48.736966 (+ 1148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21000,"mutex_wait_us":941,"thread_start_us":23653,"threads_started":1}
I20260926 02:24:48.738085 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.707783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.12.244.193:36699)
0926 02:24:48.727144 (+ 19361us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:48.727154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:48.727191 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:24:48.727309 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:48.727316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:48.727657 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:48.727877 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:48.729529 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.731177 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.733750 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.735355 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.736056 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.736267 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:48.736337 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 02:24:48.736519 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19145,"thread_start_us":19002,"threads_started":1}
I20260926 02:24:48.738368 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.704815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:41321 (local address 127.0.0.1:59372)
0926 02:24:48.728145 (+ 23330us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:48.728164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:48.728266 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:48.730430 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:48.730435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:48.730460 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:48.730807 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.730819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.733008 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.733014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.735272 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:48.735286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.737589 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.737596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:48.737789 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.737822 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 02:24:48.737948 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 02:24:48.738091 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23184,"thread_start_us":22379,"threads_started":1}
I20260926 02:24:48.738981 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:48.705513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.12.244.194:41321)
0926 02:24:48.729815 (+ 24302us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:48.729827 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:48.729857 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:24:48.729961 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:48.729969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:48.730225 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:48.731006 (+ 781us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:48.731261 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.732856 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.735522 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:48.737090 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:48.737252 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:48.737450 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:48.738490 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0926 02:24:48.738660 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24130,"thread_start_us":27465,"threads_started":1}
I20260926 02:24:48.740906 31480 tablet_service.cc:1467] Processing CreateTablet for tablet 392bcba9b0a145949fb8689ec3c30512 (DEFAULT_TABLE table=test-table [id=1c5a58983fa544f9b19d98af3d0d791b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:48.741837 31219 tablet_service.cc:1467] Processing CreateTablet for tablet 392bcba9b0a145949fb8689ec3c30512 (DEFAULT_TABLE table=test-table [id=1c5a58983fa544f9b19d98af3d0d791b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:48.742585 31480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392bcba9b0a145949fb8689ec3c30512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:48.742585 31350 tablet_service.cc:1467] Processing CreateTablet for tablet 392bcba9b0a145949fb8689ec3c30512 (DEFAULT_TABLE table=test-table [id=1c5a58983fa544f9b19d98af3d0d791b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:48.743136 31219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392bcba9b0a145949fb8689ec3c30512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:48.743916 31350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392bcba9b0a145949fb8689ec3c30512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:48.755920 31564 tablet_bootstrap.cc:492] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: Bootstrap starting.
I20260926 02:24:48.758955 31566 tablet_bootstrap.cc:492] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Bootstrap starting.
I20260926 02:24:48.760207 31564 tablet_bootstrap.cc:654] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:48.761605 31564 log.cc:824] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:48.762890 31566 tablet_bootstrap.cc:654] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:48.764499 31566 log.cc:824] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:48.766057 31565 tablet_bootstrap.cc:492] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Bootstrap starting.
I20260926 02:24:48.769093 31564 tablet_bootstrap.cc:492] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: No bootstrap required, opened a new log
I20260926 02:24:48.769434 31564 ts_tablet_manager.cc:1397] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260926 02:24:48.770153 31565 tablet_bootstrap.cc:654] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:48.771431 31566 tablet_bootstrap.cc:492] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: No bootstrap required, opened a new log
I20260926 02:24:48.771445 31565 log.cc:824] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:48.771719 31566 ts_tablet_manager.cc:1397] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 02:24:48.779906 31566 raft_consensus.cc:348] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:48.780314 31566 raft_consensus.cc:374] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:48.780362 31565 tablet_bootstrap.cc:492] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: No bootstrap required, opened a new log
I20260926 02:24:48.780467 31566 raft_consensus.cc:729] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ac9812551f4d5db5433c12c96f8cb1, State: Initialized, Role: FOLLOWER
I20260926 02:24:48.780673 31565 ts_tablet_manager.cc:1397] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260926 02:24:48.781114 31566 consensus_queue.cc:260] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:48.781510 31564 raft_consensus.cc:348] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:48.782254 31415 heartbeater.cc:503] Master 127.12.244.254:33279 was elected leader, sending a full tablet report...
I20260926 02:24:48.782790 31566 ts_tablet_manager.cc:1428] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 02:24:48.782015 31564 raft_consensus.cc:374] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:48.783277 31564 raft_consensus.cc:729] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76272099209444af80c318661cf7dbc8, State: Initialized, Role: FOLLOWER
I20260926 02:24:48.783942 31564 consensus_queue.cc:260] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:48.787030 31564 ts_tablet_manager.cc:1428] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260926 02:24:48.790930 31565 raft_consensus.cc:348] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:48.791553 31565 raft_consensus.cc:374] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:48.791673 31565 raft_consensus.cc:729] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8acfa083a0fa425abe37cb4544bf9b35, State: Initialized, Role: FOLLOWER
I20260926 02:24:48.792275 31565 consensus_queue.cc:260] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:48.793284 31546 heartbeater.cc:503] Master 127.12.244.254:33279 was elected leader, sending a full tablet report...
I20260926 02:24:48.793939 31565 ts_tablet_manager.cc:1428] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
W20260926 02:24:48.898804 31547 tablet.cc:2369] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:48.934057 31416 tablet.cc:2369] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:48.939851 31285 tablet.cc:2369] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:24:49.002144 31570 raft_consensus.cc:482] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:24:49.002506 31570 raft_consensus.cc:504] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:49.004371 31570 leader_election.cc:288] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76272099209444af80c318661cf7dbc8 (127.12.244.193:36699), 8acfa083a0fa425abe37cb4544bf9b35 (127.12.244.195:35977)
I20260926 02:24:49.012408 31572 raft_consensus.cc:482] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:24:49.012821 31572 raft_consensus.cc:504] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:49.014386 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.004936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:47717 (local address 127.12.244.193:36699)
0926 02:24:49.005122 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.005129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.006087 (+ 958us) server_negotiation.cc:407] Connection header received
0926 02:24:49.006339 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.006344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.006432 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.006535 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:49.007341 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.008503 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.011699 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.013047 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.013241 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.013314 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.013979 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.014139 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:24:49.014387 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.004907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:37829 (local address 127.12.244.195:35977)
0926 02:24:49.005123 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.005129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.006069 (+ 940us) server_negotiation.cc:407] Connection header received
0926 02:24:49.006328 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.006334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.006422 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.006638 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:49.007339 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.008478 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.011699 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.013059 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.013243 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.013314 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.014023 (+ 709us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.014152 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:24:49.014462 31574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.004709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:35977 (local address 127.12.244.194:37829)
0926 02:24:49.005890 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.005917 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:49.006106 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:49.006662 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:49.006669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:49.006723 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:49.007173 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.007181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.008663 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.008669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.011408 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.011430 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.013589 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.013596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.013806 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.013831 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:24:49.013910 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 02:24:49.014079 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":221,"spinlock_wait_cycles":37632,"thread_start_us":646,"threads_started":1}
I20260926 02:24:49.014880 31572 leader_election.cc:288] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76272099209444af80c318661cf7dbc8 (127.12.244.193:36699), 53ac9812551f4d5db5433c12c96f8cb1 (127.12.244.194:41321)
I20260926 02:24:49.015772 31573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.004709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.12.244.194:47717)
0926 02:24:49.005964 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.005980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:49.006168 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:49.006564 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:49.006570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:49.006607 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:49.007150 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.007169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.008663 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.008669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.011408 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.011430 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.013482 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.013490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.013717 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.013804 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 02:24:49.013910 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:24:49.015324 (+ 1414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":905,"spinlock_wait_cycles":27520,"thread_start_us":555,"threads_started":1}
I20260926 02:24:49.015584 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392bcba9b0a145949fb8689ec3c30512" candidate_uuid: "53ac9812551f4d5db5433c12c96f8cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acfa083a0fa425abe37cb4544bf9b35" is_pre_election: true
I20260926 02:24:49.016638 31500 raft_consensus.cc:2454] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53ac9812551f4d5db5433c12c96f8cb1 in term 0.
I20260926 02:24:49.017434 31303 leader_election.cc:302] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ac9812551f4d5db5433c12c96f8cb1, 8acfa083a0fa425abe37cb4544bf9b35; no voters:
I20260926 02:24:49.017434 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392bcba9b0a145949fb8689ec3c30512" candidate_uuid: "53ac9812551f4d5db5433c12c96f8cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76272099209444af80c318661cf7dbc8" is_pre_election: true
I20260926 02:24:49.017963 31570 raft_consensus.cc:2781] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:24:49.018121 31570 raft_consensus.cc:482] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:24:49.018042 31239 raft_consensus.cc:2454] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53ac9812551f4d5db5433c12c96f8cb1 in term 0.
I20260926 02:24:49.018230 31570 raft_consensus.cc:3037] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:49.021317 31570 raft_consensus.cc:504] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:49.022425 31239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392bcba9b0a145949fb8689ec3c30512" candidate_uuid: "53ac9812551f4d5db5433c12c96f8cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76272099209444af80c318661cf7dbc8"
I20260926 02:24:49.022662 31239 raft_consensus.cc:3037] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:49.022841 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.015225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:41321 (local address 127.12.244.195:34209)
0926 02:24:49.015743 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.015767 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:49.015896 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:49.016232 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:49.016238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:49.016264 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:49.016510 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.016524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.018658 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.018664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.020235 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.020247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.021915 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.021924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.022210 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.022247 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:24:49.022355 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:24:49.022511 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":56}
I20260926 02:24:49.022940 31570 leader_election.cc:288] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [CANDIDATE]: Term 1 election: Requested vote from peers 76272099209444af80c318661cf7dbc8 (127.12.244.193:36699), 8acfa083a0fa425abe37cb4544bf9b35 (127.12.244.195:35977)
I20260926 02:24:49.022763 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392bcba9b0a145949fb8689ec3c30512" candidate_uuid: "53ac9812551f4d5db5433c12c96f8cb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acfa083a0fa425abe37cb4544bf9b35"
I20260926 02:24:49.023207 31500 raft_consensus.cc:3037] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:49.023518 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.015698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:34209 (local address 127.12.244.194:41321)
0926 02:24:49.015905 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.015912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.015931 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:24:49.015998 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.016003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.016106 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.016218 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:49.017404 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.018527 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.020427 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.021748 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.021906 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.021972 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.023068 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.023245 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:24:49.024405 31370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392bcba9b0a145949fb8689ec3c30512" candidate_uuid: "8acfa083a0fa425abe37cb4544bf9b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53ac9812551f4d5db5433c12c96f8cb1" is_pre_election: true
I20260926 02:24:49.024927 31370 raft_consensus.cc:2379] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8acfa083a0fa425abe37cb4544bf9b35 in current term 1: Already voted for candidate 53ac9812551f4d5db5433c12c96f8cb1 in this term.
I20260926 02:24:49.026083 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.015345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.12.244.195:52583)
0926 02:24:49.016812 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.016829 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:49.016940 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:49.018167 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:49.018172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:49.018194 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:49.018527 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.018538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.020654 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.020659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.022995 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:49.023011 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.025172 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.025178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:49.025367 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.025405 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:24:49.025497 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 02:24:49.025660 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"mutex_wait_us":1,"spinlock_wait_cycles":24832,"thread_start_us":1292,"threads_started":1}
I20260926 02:24:49.026299 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.016099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:52583 (local address 127.12.244.193:36699)
0926 02:24:49.017613 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.017618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.017642 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:24:49.017705 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.017709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.017777 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.017875 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:49.019289 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.020502 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.023277 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.024717 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.024944 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.025041 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.025767 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.025913 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1372,"thread_start_us":1108,"threads_started":1}
I20260926 02:24:49.026547 31500 raft_consensus.cc:2454] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ac9812551f4d5db5433c12c96f8cb1 in term 1.
I20260926 02:24:49.027037 31303 leader_election.cc:302] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53ac9812551f4d5db5433c12c96f8cb1, 8acfa083a0fa425abe37cb4544bf9b35; no voters:
I20260926 02:24:49.027305 31239 raft_consensus.cc:2454] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ac9812551f4d5db5433c12c96f8cb1 in term 1.
I20260926 02:24:49.027071 31238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392bcba9b0a145949fb8689ec3c30512" candidate_uuid: "8acfa083a0fa425abe37cb4544bf9b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76272099209444af80c318661cf7dbc8" is_pre_election: true
I20260926 02:24:49.027446 31570 raft_consensus.cc:2781] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:49.027639 31238 raft_consensus.cc:2379] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8acfa083a0fa425abe37cb4544bf9b35 in current term 1: Already voted for candidate 53ac9812551f4d5db5433c12c96f8cb1 in this term.
I20260926 02:24:49.028100 31570 raft_consensus.cc:686] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 53ac9812551f4d5db5433c12c96f8cb1, State: Running, Role: LEADER
I20260926 02:24:49.028409 31434 leader_election.cc:302] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8acfa083a0fa425abe37cb4544bf9b35; no voters: 53ac9812551f4d5db5433c12c96f8cb1, 76272099209444af80c318661cf7dbc8
I20260926 02:24:49.028714 31570 consensus_queue.cc:237] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } }
I20260926 02:24:49.029232 31572 raft_consensus.cc:2726] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 02:24:49.034082 31097 catalog_manager.cc:5495] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53ac9812551f4d5db5433c12c96f8cb1 (127.12.244.194). New cstate: current_term: 1 leader_uuid: "53ac9812551f4d5db5433c12c96f8cb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53ac9812551f4d5db5433c12c96f8cb1" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 41321 } health_report { overall_health: HEALTHY } } }
I20260926 02:24:49.113879 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.107998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.12.244.194:41321)
0926 02:24:49.108199 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.108205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.108323 (+ 118us) server_negotiation.cc:407] Connection header received
0926 02:24:49.108618 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.108622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.108714 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.108889 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:24:49.109282 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.110067 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.111630 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.112052 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.112217 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.112732 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:24:49.113255 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:24:49.113419 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.113479 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.113562 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:24:49.122562 31238 raft_consensus.cc:1261] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 1 FOLLOWER]: Refusing update from remote peer 53ac9812551f4d5db5433c12c96f8cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:49.122596 31500 raft_consensus.cc:1261] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 1 FOLLOWER]: Refusing update from remote peer 53ac9812551f4d5db5433c12c96f8cb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:49.123601 31577 consensus_queue.cc:1035] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76272099209444af80c318661cf7dbc8" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:49.124039 31570 consensus_queue.cc:1035] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8acfa083a0fa425abe37cb4544bf9b35" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:49.136726 31580 mvcc.cc:205] Tried to move back new op lower bound from 7333435347435323392 to 7333435347066478592. Current Snapshot: MvccSnapshot[applied={T|T < 7333435347435323392}]
I20260926 02:24:49.154222 31097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:49.154506 31097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:49.158406 31097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d663432466074ca4b97a1b46d48360df: Sending DeleteTablet for 3 replicas of tablet 392bcba9b0a145949fb8689ec3c30512
I20260926 02:24:49.159775 31480 tablet_service.cc:1514] Processing DeleteTablet for tablet 392bcba9b0a145949fb8689ec3c30512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:49 UTC) from {username='slave'} at 127.0.0.1:40838
I20260926 02:24:49.160003 31350 tablet_service.cc:1514] Processing DeleteTablet for tablet 392bcba9b0a145949fb8689ec3c30512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:49 UTC) from {username='slave'} at 127.0.0.1:59372
I20260926 02:24:49.160163 31219 tablet_service.cc:1514] Processing DeleteTablet for tablet 392bcba9b0a145949fb8689ec3c30512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:49 UTC) from {username='slave'} at 127.0.0.1:51712
I20260926 02:24:49.161182 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31157
I20260926 02:24:49.173350 31591 tablet_replica.cc:330] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: stopping tablet replica
I20260926 02:24:49.175047 31591 raft_consensus.cc:2229] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:49.175843 31591 raft_consensus.cc:2258] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:49.179955 31591 ts_tablet_manager.cc:1905] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:49.180757 31592 tablet_replica.cc:330] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: stopping tablet replica
I20260926 02:24:49.180943 31576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.163247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.12.244.193:36699)
0926 02:24:49.163494 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.163504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.173229 (+ 9725us) server_negotiation.cc:407] Connection header received
0926 02:24:49.173482 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.173488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.173630 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.173769 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:49.174203 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.174945 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.176656 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.177107 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.177255 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.179134 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:49.179151 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:49.179179 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:49.179997 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:49.180063 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:49.180102 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:49.180200 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.180448 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.180578 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:24:49.181866 31594 tablet_replica.cc:330] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8: stopping tablet replica
I20260926 02:24:49.182106 31279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:49.182948 31592 raft_consensus.cc:2229] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:24:49.183676 31592 raft_consensus.cc:2258] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:49.189332 31591 ts_tablet_manager.cc:1918] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:49.189638 31591 log.cc:1196] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/392bcba9b0a145949fb8689ec3c30512
I20260926 02:24:49.190348 31591 ts_tablet_manager.cc:1939] T 392bcba9b0a145949fb8689ec3c30512 P 8acfa083a0fa425abe37cb4544bf9b35: Deleting consensus metadata
I20260926 02:24:49.191643 31592 ts_tablet_manager.cc:1905] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:49.192938 31081 catalog_manager.cc:4841] TS 8acfa083a0fa425abe37cb4544bf9b35 (127.12.244.195:35977): tablet 392bcba9b0a145949fb8689ec3c30512 (table test-table [id=1c5a58983fa544f9b19d98af3d0d791b]) successfully deleted
I20260926 02:24:49.200816 31592 ts_tablet_manager.cc:1918] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:49.201128 31592 log.cc:1196] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/392bcba9b0a145949fb8689ec3c30512
I20260926 02:24:49.201784 31592 ts_tablet_manager.cc:1939] T 392bcba9b0a145949fb8689ec3c30512 P 53ac9812551f4d5db5433c12c96f8cb1: Deleting consensus metadata
I20260926 02:24:49.203794 31084 catalog_manager.cc:4841] TS 53ac9812551f4d5db5433c12c96f8cb1 (127.12.244.194:41321): tablet 392bcba9b0a145949fb8689ec3c30512 (table test-table [id=1c5a58983fa544f9b19d98af3d0d791b]) successfully deleted
I20260926 02:24:49.182497 31594 raft_consensus.cc:2229] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:49.296895 31594 raft_consensus.cc:2258] T 392bcba9b0a145949fb8689ec3c30512 P 76272099209444af80c318661cf7dbc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:49.297145 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31157
W20260926 02:24:49.315866 31082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.244.193:36699 (error 108)
I20260926 02:24:49.316593 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31288
I20260926 02:24:49.319133 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.317491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51728)
0926 02:24:49.318725 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.318881 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1103,"thread_start_us":1002,"threads_started":1}
W20260926 02:24:49.320061 31082 catalog_manager.cc:4584] TS 76272099209444af80c318661cf7dbc8 (127.12.244.193:36699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 392bcba9b0a145949fb8689ec3c30512: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
I20260926 02:24:49.324705 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.317708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.12.244.194:41321)
0926 02:24:49.317890 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.317896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.317917 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:24:49.317999 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.318006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.318141 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.318254 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:49.318818 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.319732 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.321272 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.321710 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.321848 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.323086 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:49.323214 (+ 128us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:49.323232 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:49.323869 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:49.323943 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:49.323965 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:49.324094 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.324264 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.324374 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:24:49.325497 31410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:49.374040 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.373493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51740)
0926 02:24:49.373744 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.373876 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 02:24:49.375339 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.374834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51744)
0926 02:24:49.375046 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.375183 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 02:24:49.435993 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.435427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51754)
0926 02:24:49.435670 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.435786 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:24:49.437127 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.436678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51760)
0926 02:24:49.436859 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.436945 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:24:49.440572 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31288
I20260926 02:24:49.458988 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31418
I20260926 02:24:49.466696 31561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.459860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.12.244.195:35977)
0926 02:24:49.460059 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:49.460065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:49.460171 (+ 106us) server_negotiation.cc:407] Connection header received
0926 02:24:49.460396 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:49.460400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:49.460564 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:49.460696 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:49.461120 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.461863 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.463330 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:49.463781 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:49.463925 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:49.465239 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:49.465248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:49.465264 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:49.465896 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:49.465945 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:49.465965 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:49.466112 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:49.466237 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 02:24:49.466352 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:24:49.467607 31541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:49.517970 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.517353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51772)
0926 02:24:49.517667 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.517790 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260926 02:24:49.519335 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.518778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51776)
0926 02:24:49.519018 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.519162 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 02:24:49.561147 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31418
I20260926 02:24:49.578145 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31066
I20260926 02:24:49.579275 31128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:49.692374 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:49.691736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:36699 (local address 127.0.0.1:51792)
0926 02:24:49.691998 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:49.692185 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:36699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:24:49.692619 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31066
2026-09-26T02:24:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3140 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:24:49 dist-test-slave-qfmw krb5kdc[31604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:24:49 dist-test-slave-qfmw krb5kdc[31604](info): set up 2 sockets
Sep 26 02:24:49 dist-test-slave-qfmw krb5kdc[31604](info): commencing operation
krb5kdc: starting...
W20260926 02:24:50.825742 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:24:50 dist-test-slave-qfmw krb5kdc[31604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:24:51.091506 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34865
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:34865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:51.445399 31617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:51.445791 31617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:51.445945 31617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:51.453480 31617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:24:51.453661 31617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:51.453771 31617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:51.453874 31617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:24:51.453949 31617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:24:51.464985 31617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:34865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:51.466440 31617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:51.467912 31617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:51.480365 31622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:51.480315 31625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:51.480404 31623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:51.481812 31617 server_base.cc:1034] running on GCE node
I20260926 02:24:51.482758 31617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:51.484368 31617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:51.485606 31617 hybrid_clock.cc:648] HybridClock initialized: now 1790389491485558 us; error 31 us; skew 500 ppm
I20260926 02:24:51.485922 31617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:51 dist-test-slave-qfmw krb5kdc[31604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389491, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:51.489198 31617 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:24:51.490845 31617 webserver.cc:460] Webserver started at http://127.12.244.254:39045/ using document root <none> and password file <none>
I20260926 02:24:51.491613 31617 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:51.491722 31617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:51.491978 31617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:51.495285 31617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "b73eee9b9a084cc88d79dcdf4ab01dce"
format_stamp: "Formatted at 2026-09-26 02:24:51 on dist-test-slave-qfmw"
server_key: "ede3610a82d296db48e0975187aade8b"
server_key_iv: "d4cc9ca2ad6d9f43e716b557c129d495"
server_key_version: "encryptionkey@0"
I20260926 02:24:51.496060 31617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "b73eee9b9a084cc88d79dcdf4ab01dce"
format_stamp: "Formatted at 2026-09-26 02:24:51 on dist-test-slave-qfmw"
server_key: "ede3610a82d296db48e0975187aade8b"
server_key_iv: "d4cc9ca2ad6d9f43e716b557c129d495"
server_key_version: "encryptionkey@0"
I20260926 02:24:51.501813 31617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 02:24:51.505858 31632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:51.507655 31617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:24:51.507831 31617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "b73eee9b9a084cc88d79dcdf4ab01dce"
format_stamp: "Formatted at 2026-09-26 02:24:51 on dist-test-slave-qfmw"
server_key: "ede3610a82d296db48e0975187aade8b"
server_key_iv: "d4cc9ca2ad6d9f43e716b557c129d495"
server_key_version: "encryptionkey@0"
I20260926 02:24:51.508025 31617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:51.531525 31617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:51.536325 31617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:51.536597 31617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:51.557021 31617 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:34865
I20260926 02:24:51.557103 31684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:34865 every 8 connection(s)
I20260926 02:24:51.558581 31617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:24:51.562776 31685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:51.568328 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31617
I20260926 02:24:51.568547 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:24:51.569005 13267 external_mini_cluster.cc:1428] Setting key c7c94b20a8f8bcf162cabd7bad80f4a1
I20260926 02:24:51.576011 31685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce: Bootstrap starting.
I20260926 02:24:51.580133 31685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:51.581300 31685 log.cc:824] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:51.584973 31685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce: No bootstrap required, opened a new log
Sep 26 02:24:51 dist-test-slave-qfmw krb5kdc[31604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:51.596012 31685 raft_consensus.cc:348] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34865 } }
I20260926 02:24:51.596526 31685 raft_consensus.cc:374] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:51.596562 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:51.571185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.12.244.254:34865)
0926 02:24:51.572096 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:51.572129 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:51.572187 (+ 58us) server_negotiation.cc:407] Connection header received
0926 02:24:51.574750 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:51.574784 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:51.575369 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:51.575864 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:51.577440 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:51.579574 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:51.581071 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:51.582223 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:51.582370 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:51.587064 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:51.587076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:51.587118 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:51.587175 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:51.591497 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:51.592287 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:51.592297 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:51.592308 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:51.592558 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:51.592919 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:51.592924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:51.592928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:51.593787 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:51.594104 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:51.594545 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 02:24:51.595070 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":379,"threads_started":1}
I20260926 02:24:51.596671 31685 raft_consensus.cc:729] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73eee9b9a084cc88d79dcdf4ab01dce, State: Initialized, Role: FOLLOWER
I20260926 02:24:51.597303 31685 consensus_queue.cc:260] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34865 } }
I20260926 02:24:51.597576 31685 raft_consensus.cc:388] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:24:51.597733 31685 raft_consensus.cc:482] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:24:51.597925 31685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:51.600649 31685 raft_consensus.cc:504] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34865 } }
I20260926 02:24:51.601227 31685 leader_election.cc:302] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b73eee9b9a084cc88d79dcdf4ab01dce; no voters:
I20260926 02:24:51.601847 31685 leader_election.cc:288] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:24:51.601999 31690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:51.603363 31690 raft_consensus.cc:686] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [term 1 LEADER]: Becoming Leader. State: Replica: b73eee9b9a084cc88d79dcdf4ab01dce, State: Running, Role: LEADER
I20260926 02:24:51.603987 31690 consensus_queue.cc:237] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34865 } }
I20260926 02:24:51.604218 31685 sys_catalog.cc:564] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:24:51.607052 31692 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b73eee9b9a084cc88d79dcdf4ab01dce. Latest consensus state: current_term: 1 leader_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34865 } } }
I20260926 02:24:51.607545 31692 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:51.607997 31691 sys_catalog.cc:455] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b73eee9b9a084cc88d79dcdf4ab01dce" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34865 } } }
I20260926 02:24:51.608372 31691 sys_catalog.cc:458] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:51.610952 31699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:24:51.617017 31699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:24:51.627514 31699 catalog_manager.cc:1347] Generated new cluster ID: 938c01ff8be3412bb6fd8c548a069c67
I20260926 02:24:51.627668 31699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:24:51.656445 31699 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:24:51.657910 31699 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:24:51.668897 31699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce: Generated new TSK 0
I20260926 02:24:51.670148 31699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:24:51.742458 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34865
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:52.093047 31711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:52.093415 31711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:52.093580 31711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:52.102528 31711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:52.102710 31711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:52.102892 31711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:24:52.114243 31711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:52.115779 31711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:52.117197 31711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:52.131784 31717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:52.132351 31716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:52.132699 31719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:52.132864 31711 server_base.cc:1034] running on GCE node
I20260926 02:24:52.133597 31711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:52.134833 31711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:52.136126 31711 hybrid_clock.cc:648] HybridClock initialized: now 1790389492136038 us; error 76 us; skew 500 ppm
I20260926 02:24:52.136504 31711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:52 dist-test-slave-qfmw krb5kdc[31604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:52.140091 31711 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:24:52.141768 31711 webserver.cc:460] Webserver started at http://127.12.244.193:40445/ using document root <none> and password file <none>
I20260926 02:24:52.142588 31711 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:52.142748 31711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:52.143167 31711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:52.146494 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "66410791f1cc478eba8273375a3275ad"
format_stamp: "Formatted at 2026-09-26 02:24:52 on dist-test-slave-qfmw"
server_key: "2b11d1f59e4b77f1c6a6d6690d8673f4"
server_key_iv: "2068464c37f1be3326bf21bff0cc5a5b"
server_key_version: "encryptionkey@0"
I20260926 02:24:52.147459 31711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "66410791f1cc478eba8273375a3275ad"
format_stamp: "Formatted at 2026-09-26 02:24:52 on dist-test-slave-qfmw"
server_key: "2b11d1f59e4b77f1c6a6d6690d8673f4"
server_key_iv: "2068464c37f1be3326bf21bff0cc5a5b"
server_key_version: "encryptionkey@0"
I20260926 02:24:52.153630 31711 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:24:52.157979 31726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:52.159699 31711 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 02:24:52.159922 31711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "66410791f1cc478eba8273375a3275ad"
format_stamp: "Formatted at 2026-09-26 02:24:52 on dist-test-slave-qfmw"
server_key: "2b11d1f59e4b77f1c6a6d6690d8673f4"
server_key_iv: "2068464c37f1be3326bf21bff0cc5a5b"
server_key_version: "encryptionkey@0"
I20260926 02:24:52.160187 31711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:52.175064 31711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:52.180130 31711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:52.180460 31711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:52.181542 31711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:52.183024 31711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:52.183172 31711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 02:24:52.183326 31711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:52.183404 31711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:52.219589 31711 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:45099
I20260926 02:24:52.219630 31839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:45099 every 8 connection(s)
I20260926 02:24:52.221011 31711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:24:52.228163 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31711
I20260926 02:24:52.228361 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:24:52.228932 13267 external_mini_cluster.cc:1428] Setting key 013bfbdfb4615ddbec8cfc4327ac59de
Sep 26 02:24:52 dist-test-slave-qfmw krb5kdc[31604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:52.246107 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:52.223763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:55153 (local address 127.12.244.254:34865)
0926 02:24:52.224248 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:52.224252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:52.225108 (+ 856us) server_negotiation.cc:407] Connection header received
0926 02:24:52.226997 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:52.227003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:52.227165 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:52.227284 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:52.229484 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.230592 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.233340 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.233812 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.233934 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:52.239122 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:52.239133 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:52.239142 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:52.239198 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:52.242287 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:52.243480 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:52.243485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:52.243487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:52.243631 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:52.244194 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:52.244197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:52.244199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:52.244491 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:52.244640 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:52.245625 (+ 985us) server_negotiation.cc:299] Negotiation successful
0926 02:24:52.245827 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":251,"threads_started":1}
I20260926 02:24:52.247754 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:52.224196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34865 (local address 127.12.244.193:55153)
0926 02:24:52.224930 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:52.224968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:52.226699 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:52.227549 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:52.227559 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:52.228375 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:52.229269 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:52.229299 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.230897 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.230902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:52.233132 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:52.233155 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.234086 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.234096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:52.234464 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:52.235411 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:52.235449 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:52.238745 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:52.242500 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:52.242515 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:52.242538 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:52.243316 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:52.243756 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:52.243764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:52.243770 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:52.244070 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:52.244627 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:52.244638 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:52.245180 (+ 542us) client_negotiation.cc:769] Sending connection context
0926 02:24:52.245504 (+ 324us) client_negotiation.cc:240] Negotiation successful
0926 02:24:52.246124 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":275,"threads_started":1}
I20260926 02:24:52.249712 31840 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34865
I20260926 02:24:52.250084 31840 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:52.250905 31840 heartbeater.cc:511] Master 127.12.244.254:34865 requested a full tablet report, sending...
I20260926 02:24:52.253427 31649 ts_manager.cc:194] Registered new tserver with Master: 66410791f1cc478eba8273375a3275ad (127.12.244.193:45099)
I20260926 02:24:52.254972 31649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:55153
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:24:52.291070 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34865
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:52.648869 31846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:52.649222 31846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:52.649407 31846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:52.657621 31846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:52.657790 31846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:52.658023 31846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:24:52.668900 31846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:52.670553 31846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:52.672637 31846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:52.687017 31854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:52.687142 31852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:52.687767 31851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:52.687835 31846 server_base.cc:1034] running on GCE node
I20260926 02:24:52.688542 31846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:52.689551 31846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:52.690773 31846 hybrid_clock.cc:648] HybridClock initialized: now 1790389492690729 us; error 41 us; skew 500 ppm
I20260926 02:24:52.691077 31846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:52 dist-test-slave-qfmw krb5kdc[31604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:52.694908 31846 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:24:52.699724 31846 webserver.cc:460] Webserver started at http://127.12.244.194:33705/ using document root <none> and password file <none>
I20260926 02:24:52.700857 31846 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:52.701053 31846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:52.701540 31846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:52.705957 31846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71"
format_stamp: "Formatted at 2026-09-26 02:24:52 on dist-test-slave-qfmw"
server_key: "790be7ce4b986e05652b0442af5f5a9e"
server_key_iv: "1ac5930940c7197844c6cbc8345fb8fd"
server_key_version: "encryptionkey@0"
I20260926 02:24:52.706885 31846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71"
format_stamp: "Formatted at 2026-09-26 02:24:52 on dist-test-slave-qfmw"
server_key: "790be7ce4b986e05652b0442af5f5a9e"
server_key_iv: "1ac5930940c7197844c6cbc8345fb8fd"
server_key_version: "encryptionkey@0"
I20260926 02:24:52.713584 31846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:24:52.717839 31861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:52.719720 31846 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:24:52.719954 31846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71"
format_stamp: "Formatted at 2026-09-26 02:24:52 on dist-test-slave-qfmw"
server_key: "790be7ce4b986e05652b0442af5f5a9e"
server_key_iv: "1ac5930940c7197844c6cbc8345fb8fd"
server_key_version: "encryptionkey@0"
I20260926 02:24:52.720212 31846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:52.734323 31846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:52.739234 31846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:52.739550 31846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:52.740607 31846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:52.742079 31846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:52.742188 31846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:52.742339 31846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:52.742413 31846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:52.777989 31846 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:37819
I20260926 02:24:52.778043 31974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:37819 every 8 connection(s)
I20260926 02:24:52.779516 31846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:24:52.789251 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31846
I20260926 02:24:52.789465 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:24:52.790143 13267 external_mini_cluster.cc:1428] Setting key 5321cde461b2442f4f012e68857570b4
Sep 26 02:24:52 dist-test-slave-qfmw krb5kdc[31604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389492, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:52.803377 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:52.782335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:45283 (local address 127.12.244.254:34865)
0926 02:24:52.782841 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:52.782847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:52.783900 (+ 1053us) server_negotiation.cc:407] Connection header received
0926 02:24:52.785829 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:52.785834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:52.785912 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:52.786018 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:52.787917 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.788691 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.791231 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.791793 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.791902 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:52.795746 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:52.795756 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:52.795763 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:52.795812 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:52.799575 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:52.800432 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:52.800442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:52.800448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:52.800647 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:52.801331 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:52.801335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:52.801337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:52.801685 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:52.801815 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:52.802833 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0926 02:24:52.803019 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260926 02:24:52.805073 31978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:52.782923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34865 (local address 127.12.244.194:45283)
0926 02:24:52.783719 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:52.783768 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:52.785505 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:52.786320 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:52.786330 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:52.786935 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:52.787706 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:52.787728 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.788901 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.788909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:52.791007 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:52.791029 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:52.792053 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:52.792061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:52.792317 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:52.793080 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:52.793105 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:52.795428 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:52.799750 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:52.799759 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:52.799774 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:52.800270 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:52.800848 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:52.800854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:52.800860 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:52.801182 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:52.801831 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:52.801842 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:52.802356 (+ 514us) client_negotiation.cc:769] Sending connection context
0926 02:24:52.802740 (+ 384us) client_negotiation.cc:240] Negotiation successful
0926 02:24:52.803423 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":325,"threads_started":1}
I20260926 02:24:52.806833 31975 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34865
I20260926 02:24:52.807265 31975 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:52.808139 31975 heartbeater.cc:511] Master 127.12.244.254:34865 requested a full tablet report, sending...
I20260926 02:24:52.809940 31649 ts_manager.cc:194] Registered new tserver with Master: 23b1b2e6ed0f416b8a29d9c2a5d56d71 (127.12.244.194:37819)
I20260926 02:24:52.811293 31649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:45283
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:24:52.851797 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34865
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:53.203375 31981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:53.203701 31981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:53.203796 31981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:53.211519 31981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:53.211705 31981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:53.211885 31981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:24:53.222286 31981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:34865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:53.223774 31981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:53.225322 31981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:53.239672 31987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:53.239763 31989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:53.239755 31986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:53.240906 31981 server_base.cc:1034] running on GCE node
I20260926 02:24:53.241675 31981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:53.242764 31981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:53.244009 31981 hybrid_clock.cc:648] HybridClock initialized: now 1790389493243945 us; error 76 us; skew 500 ppm
I20260926 02:24:53.244355 31981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:53 dist-test-slave-qfmw krb5kdc[31604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389493, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:53.247939 31981 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:24:53.249747 31981 webserver.cc:460] Webserver started at http://127.12.244.195:34423/ using document root <none> and password file <none>
I20260926 02:24:53.250530 31981 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:53.250686 31981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:53.251039 31981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:53.254314 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "ab7009655c3443efa6333ae6f2152de6"
format_stamp: "Formatted at 2026-09-26 02:24:53 on dist-test-slave-qfmw"
server_key: "10e722fe261430c3c51f0af5bc39dd94"
server_key_iv: "7685920bc5ce03070ef0b7fb43672b18"
server_key_version: "encryptionkey@0"
I20260926 02:24:53.255302 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "ab7009655c3443efa6333ae6f2152de6"
format_stamp: "Formatted at 2026-09-26 02:24:53 on dist-test-slave-qfmw"
server_key: "10e722fe261430c3c51f0af5bc39dd94"
server_key_iv: "7685920bc5ce03070ef0b7fb43672b18"
server_key_version: "encryptionkey@0"
I20260926 02:24:53.258539 31840 heartbeater.cc:503] Master 127.12.244.254:34865 was elected leader, sending a full tablet report...
I20260926 02:24:53.261700 31981 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 02:24:53.266320 31996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:53.268239 31981 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 02:24:53.268472 31981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "ab7009655c3443efa6333ae6f2152de6"
format_stamp: "Formatted at 2026-09-26 02:24:53 on dist-test-slave-qfmw"
server_key: "10e722fe261430c3c51f0af5bc39dd94"
server_key_iv: "7685920bc5ce03070ef0b7fb43672b18"
server_key_version: "encryptionkey@0"
I20260926 02:24:53.268731 31981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:53.284603 31981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:53.289316 31981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:53.289810 31981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:53.291054 31981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:53.292732 31981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:53.292845 31981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:53.293015 31981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:53.293100 31981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:53.332294 31981 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:44439
I20260926 02:24:53.332327 32108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:44439 every 8 connection(s)
I20260926 02:24:53.333838 31981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:24:53.338028 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 31981
I20260926 02:24:53.338231 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:24:53.338755 13267 external_mini_cluster.cc:1428] Setting key 3acd08d40c3e1ae9ef3520df9613f7be
Sep 26 02:24:53 dist-test-slave-qfmw krb5kdc[31604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389493, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:53.354970 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.336774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:58571 (local address 127.12.244.254:34865)
0926 02:24:53.337305 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.337311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.338119 (+ 808us) server_negotiation.cc:407] Connection header received
0926 02:24:53.340369 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.340374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.340482 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.340602 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:53.342163 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.342935 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.344569 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.344982 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.345122 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.348617 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:53.348622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:53.348625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:53.348657 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:53.351668 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.352501 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.352505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.352507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.352639 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.353115 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.353120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.353122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.353405 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:53.353581 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.354322 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.354569 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":231,"threads_started":1}
I20260926 02:24:53.355964 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.337166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:34865 (local address 127.12.244.195:58571)
0926 02:24:53.337939 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:53.337977 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:53.340037 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:53.340878 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:53.340887 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:53.341419 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:53.341994 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.342011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.343129 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.343134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.344379 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.344391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.345226 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.345235 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.345476 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.346122 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:53.346144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:53.348386 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:53.351836 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:53.351846 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:53.351866 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:53.352342 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:53.352755 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:53.352763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:53.352766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:53.352981 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:53.353549 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:53.353558 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:53.353950 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 02:24:53.354218 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 02:24:53.354786 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":331,"threads_started":1}
I20260926 02:24:53.357630 32109 heartbeater.cc:347] Connected to a master server at 127.12.244.254:34865
I20260926 02:24:53.357980 32109 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:53.358724 32109 heartbeater.cc:511] Master 127.12.244.254:34865 requested a full tablet report, sending...
I20260926 02:24:53.360587 31649 ts_manager.cc:194] Registered new tserver with Master: ab7009655c3443efa6333ae6f2152de6 (127.12.244.195:44439)
I20260926 02:24:53.361464 31649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:58571
I20260926 02:24:53.366997 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:24:53.386981 32111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.371986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.12.244.254:34865)
0926 02:24:53.372234 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.372238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.372855 (+ 617us) server_negotiation.cc:407] Connection header received
0926 02:24:53.373016 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.373020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.373090 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.373184 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:53.374919 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.375853 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.377326 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.377757 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.377870 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.381108 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:53.381113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:53.381117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:53.381158 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:53.384142 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.385073 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.385077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.385079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.385213 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.385716 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.385720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.385722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.386062 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:53.386267 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.386611 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.386757 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260926 02:24:53.391434 31649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:24:53.398674 31649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:24:53.439492 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.425404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.12.244.195:44439)
0926 02:24:53.431443 (+ 6039us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.431455 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.431502 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:24:53.431588 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.431594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.431775 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.432043 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:53.432670 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.434298 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.436868 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.438213 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.438420 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.438628 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.438904 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.439072 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5829,"thread_start_us":5595,"threads_started":1}
I20260926 02:24:53.439628 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.425041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44439 (local address 127.0.0.1:57020)
0926 02:24:53.429309 (+ 4268us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:53.429333 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:53.429498 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:53.432023 (+ 2525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:53.432027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:53.432055 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:53.432411 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.432446 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.434467 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.434478 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.436629 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.436646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.438418 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.438424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.438633 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.438699 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 02:24:53.438850 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 02:24:53.439057 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4089,"mutex_wait_us":208,"thread_start_us":3866,"threads_started":1}
I20260926 02:24:53.440523 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.413133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.12.244.193:45099)
0926 02:24:53.428395 (+ 15262us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.428410 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.428719 (+ 309us) server_negotiation.cc:407] Connection header received
0926 02:24:53.429637 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.429648 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.429970 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.430232 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:53.431932 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.433573 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.437050 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.438703 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.438921 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.439147 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.439987 (+ 840us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.440210 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15004,"thread_start_us":15285,"threads_started":1}
I20260926 02:24:53.440582 32122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.412391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45099 (local address 127.0.0.1:34776)
0926 02:24:53.428511 (+ 16120us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:53.428543 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:53.428758 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:53.431213 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:53.431218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:53.431245 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:53.431684 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.431700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.433761 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.433766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.436809 (+ 3043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.436827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.439421 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.439427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.439737 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.439782 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:24:53.439880 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 02:24:53.440061 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15910,"mutex_wait_us":252,"thread_start_us":11524,"threads_started":1}
I20260926 02:24:53.440783 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.412098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:37819 (local address 127.0.0.1:48154)
0926 02:24:53.424376 (+ 12278us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:53.424425 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:53.424699 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:53.430626 (+ 5927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:53.430635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:53.430680 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:53.431242 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.431259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.434429 (+ 3170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.434435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.437126 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.437143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.439408 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.439416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.440068 (+ 652us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.440106 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:24:53.440196 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 02:24:53.440333 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11986,"mutex_wait_us":450,"thread_start_us":16806,"threads_started":1}
I20260926 02:24:53.443317 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.412486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.12.244.194:37819)
0926 02:24:53.423784 (+ 11298us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.423798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.424593 (+ 795us) server_negotiation.cc:407] Connection header received
0926 02:24:53.430081 (+ 5488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.430096 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.430381 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.430604 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:53.431462 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.434261 (+ 2799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.437341 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.438836 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.439026 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.439200 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.441198 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.441354 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11034,"thread_start_us":10848,"threads_started":1}
I20260926 02:24:53.444914 32044 tablet_service.cc:1467] Processing CreateTablet for tablet d69e5c6df9f7408d83e21d77e3e80aab (DEFAULT_TABLE table=test-table [id=830e6afb5ead4b879c277912c656c80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:53.444919 31774 tablet_service.cc:1467] Processing CreateTablet for tablet d69e5c6df9f7408d83e21d77e3e80aab (DEFAULT_TABLE table=test-table [id=830e6afb5ead4b879c277912c656c80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:53.446695 32044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69e5c6df9f7408d83e21d77e3e80aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:53.446692 31774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69e5c6df9f7408d83e21d77e3e80aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:53.446688 31909 tablet_service.cc:1467] Processing CreateTablet for tablet d69e5c6df9f7408d83e21d77e3e80aab (DEFAULT_TABLE table=test-table [id=830e6afb5ead4b879c277912c656c80a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:53.448275 31909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d69e5c6df9f7408d83e21d77e3e80aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:53.460311 32128 tablet_bootstrap.cc:492] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Bootstrap starting.
I20260926 02:24:53.464788 32128 tablet_bootstrap.cc:654] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:53.466004 32128 log.cc:824] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:53.469594 32128 tablet_bootstrap.cc:492] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: No bootstrap required, opened a new log
I20260926 02:24:53.469926 32128 ts_tablet_manager.cc:1397] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:24:53.475811 32129 tablet_bootstrap.cc:492] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Bootstrap starting.
I20260926 02:24:53.476125 32130 tablet_bootstrap.cc:492] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Bootstrap starting.
I20260926 02:24:53.480041 32130 tablet_bootstrap.cc:654] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:53.480037 32129 tablet_bootstrap.cc:654] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:53.481480 32129 log.cc:824] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:53.481480 32130 log.cc:824] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:53.483537 32128 raft_consensus.cc:348] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.484072 32128 raft_consensus.cc:374] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:53.484176 32128 raft_consensus.cc:729] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b1b2e6ed0f416b8a29d9c2a5d56d71, State: Initialized, Role: FOLLOWER
I20260926 02:24:53.485056 32128 consensus_queue.cc:260] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.485976 32129 tablet_bootstrap.cc:492] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: No bootstrap required, opened a new log
I20260926 02:24:53.485976 32130 tablet_bootstrap.cc:492] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: No bootstrap required, opened a new log
I20260926 02:24:53.486310 32129 ts_tablet_manager.cc:1397] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260926 02:24:53.486419 31975 heartbeater.cc:503] Master 127.12.244.254:34865 was elected leader, sending a full tablet report...
I20260926 02:24:53.486961 32128 ts_tablet_manager.cc:1428] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260926 02:24:53.487293 32130 ts_tablet_manager.cc:1397] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.002s
I20260926 02:24:53.495712 32129 raft_consensus.cc:348] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.496218 32129 raft_consensus.cc:374] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:53.496331 32129 raft_consensus.cc:729] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66410791f1cc478eba8273375a3275ad, State: Initialized, Role: FOLLOWER
I20260926 02:24:53.497088 32129 consensus_queue.cc:260] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.497078 32130 raft_consensus.cc:348] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.497485 32130 raft_consensus.cc:374] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:53.497568 32130 raft_consensus.cc:729] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7009655c3443efa6333ae6f2152de6, State: Initialized, Role: FOLLOWER
I20260926 02:24:53.498255 32130 consensus_queue.cc:260] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.499172 32129 ts_tablet_manager.cc:1428] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260926 02:24:53.499356 32109 heartbeater.cc:503] Master 127.12.244.254:34865 was elected leader, sending a full tablet report...
I20260926 02:24:53.500100 32130 ts_tablet_manager.cc:1428] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260926 02:24:53.531809 31976 tablet.cc:2369] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:24:53.542692 32136 raft_consensus.cc:482] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:24:53.543022 32136 raft_consensus.cc:504] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.544946 32136 leader_election.cc:288] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b1b2e6ed0f416b8a29d9c2a5d56d71 (127.12.244.194:37819), 66410791f1cc478eba8273375a3275ad (127.12.244.193:45099)
I20260926 02:24:53.551491 32112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.545175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:37819 (local address 127.12.244.195:57155)
0926 02:24:53.545412 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:53.545431 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:53.545561 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:53.546109 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:53.546114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:53.546142 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:53.546342 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.546356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.547649 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.547655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.549498 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.549514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.550703 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.550709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.550893 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.550918 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 02:24:53.551023 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:24:53.551208 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:24:53.551491 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.545504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:57155 (local address 127.12.244.194:37819)
0926 02:24:53.545735 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.545741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.545759 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:24:53.545872 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.545877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.545957 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.546079 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:53.546499 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.547506 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.549681 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.550522 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.550711 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.550791 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.551062 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.551230 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:24:53.552583 31929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69e5c6df9f7408d83e21d77e3e80aab" candidate_uuid: "ab7009655c3443efa6333ae6f2152de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" is_pre_election: true
I20260926 02:24:53.552855 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.545736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:36061 (local address 127.12.244.193:45099)
0926 02:24:53.545960 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.545966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.546785 (+ 819us) server_negotiation.cc:407] Connection header received
0926 02:24:53.546939 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.546946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.547022 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.547197 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:53.547840 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.548919 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.550774 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.551914 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.552105 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.552196 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.552477 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.552607 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:24:53.552910 32137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.545786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:45099 (local address 127.12.244.195:36061)
0926 02:24:53.546670 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:53.546688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:53.546808 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:53.547270 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:53.547273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:53.547297 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:53.547648 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.547663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.549097 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.549101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.550568 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:53.550581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.552137 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.552143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:53.552308 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.552331 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:24:53.552439 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:24:53.552577 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":456,"threads_started":1}
I20260926 02:24:53.553193 31929 raft_consensus.cc:2454] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7009655c3443efa6333ae6f2152de6 in term 0.
I20260926 02:24:53.554056 32000 leader_election.cc:302] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b1b2e6ed0f416b8a29d9c2a5d56d71, ab7009655c3443efa6333ae6f2152de6; no voters:
I20260926 02:24:53.554028 31794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69e5c6df9f7408d83e21d77e3e80aab" candidate_uuid: "ab7009655c3443efa6333ae6f2152de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66410791f1cc478eba8273375a3275ad" is_pre_election: true
I20260926 02:24:53.554574 32136 raft_consensus.cc:2781] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:24:53.554591 31794 raft_consensus.cc:2454] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7009655c3443efa6333ae6f2152de6 in term 0.
I20260926 02:24:53.554713 32136 raft_consensus.cc:482] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:24:53.554842 32136 raft_consensus.cc:3037] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:53.557785 32136 raft_consensus.cc:504] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.558504 32136 leader_election.cc:288] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [CANDIDATE]: Term 1 election: Requested vote from peers 23b1b2e6ed0f416b8a29d9c2a5d56d71 (127.12.244.194:37819), 66410791f1cc478eba8273375a3275ad (127.12.244.193:45099)
I20260926 02:24:53.559147 31794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69e5c6df9f7408d83e21d77e3e80aab" candidate_uuid: "ab7009655c3443efa6333ae6f2152de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66410791f1cc478eba8273375a3275ad"
I20260926 02:24:53.559175 31929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d69e5c6df9f7408d83e21d77e3e80aab" candidate_uuid: "ab7009655c3443efa6333ae6f2152de6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71"
I20260926 02:24:53.559393 31929 raft_consensus.cc:3037] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:53.559415 31794 raft_consensus.cc:3037] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:53.561913 31794 raft_consensus.cc:2454] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7009655c3443efa6333ae6f2152de6 in term 1.
I20260926 02:24:53.562547 31998 leader_election.cc:302] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66410791f1cc478eba8273375a3275ad, ab7009655c3443efa6333ae6f2152de6; no voters:
I20260926 02:24:53.562819 31929 raft_consensus.cc:2454] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7009655c3443efa6333ae6f2152de6 in term 1.
I20260926 02:24:53.562948 32136 raft_consensus.cc:2781] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:53.563527 32136 raft_consensus.cc:686] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 1 LEADER]: Becoming Leader. State: Replica: ab7009655c3443efa6333ae6f2152de6, State: Running, Role: LEADER
I20260926 02:24:53.564213 32136 consensus_queue.cc:237] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } }
I20260926 02:24:53.569913 31648 catalog_manager.cc:5495] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7009655c3443efa6333ae6f2152de6 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "ab7009655c3443efa6333ae6f2152de6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7009655c3443efa6333ae6f2152de6" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44439 } health_report { overall_health: HEALTHY } } }
W20260926 02:24:53.585963 32110 tablet.cc:2369] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:24:53.602909 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.596152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.12.244.195:44439)
0926 02:24:53.596402 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.596417 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.596479 (+ 62us) server_negotiation.cc:407] Connection header received
0926 02:24:53.596829 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.596833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.596922 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.597076 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:24:53.597591 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.598416 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.600378 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.600875 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.601005 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.601627 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:24:53.602167 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:24:53.602321 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.602545 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.602601 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 02:24:53.612591 31794 raft_consensus.cc:1261] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 1 FOLLOWER]: Refusing update from remote peer ab7009655c3443efa6333ae6f2152de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:53.612617 31929 raft_consensus.cc:1261] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 1 FOLLOWER]: Refusing update from remote peer ab7009655c3443efa6333ae6f2152de6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:53.613739 32136 consensus_queue.cc:1035] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66410791f1cc478eba8273375a3275ad" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:53.614120 32138 consensus_queue.cc:1035] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b1b2e6ed0f416b8a29d9c2a5d56d71" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:53.628135 32141 mvcc.cc:205] Tried to move back new op lower bound from 7333435365823651840 to 7333435365644111872. Current Snapshot: MvccSnapshot[applied={T|T < 7333435365823651840}]
I20260926 02:24:53.646632 31649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:53.646950 31649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:53.650956 31649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b73eee9b9a084cc88d79dcdf4ab01dce: Sending DeleteTablet for 3 replicas of tablet d69e5c6df9f7408d83e21d77e3e80aab
I20260926 02:24:53.652374 31909 tablet_service.cc:1514] Processing DeleteTablet for tablet d69e5c6df9f7408d83e21d77e3e80aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:53 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:48154
I20260926 02:24:53.652539 32044 tablet_service.cc:1514] Processing DeleteTablet for tablet d69e5c6df9f7408d83e21d77e3e80aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:53 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:57020
I20260926 02:24:53.653282 31774 tablet_service.cc:1514] Processing DeleteTablet for tablet d69e5c6df9f7408d83e21d77e3e80aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:53 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:34776
I20260926 02:24:53.654109 32151 tablet_replica.cc:330] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: stopping tablet replica
I20260926 02:24:53.654282 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31711
I20260926 02:24:53.654606 32151 raft_consensus.cc:2229] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:53.655228 32151 raft_consensus.cc:2258] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:53.659982 32153 tablet_replica.cc:330] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: stopping tablet replica
I20260926 02:24:53.661114 32151 ts_tablet_manager.cc:1905] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:53.661146 32153 raft_consensus.cc:2229] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:53.661928 32153 raft_consensus.cc:2258] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:53.670038 32152 tablet_replica.cc:330] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: stopping tablet replica
I20260926 02:24:53.670575 32151 ts_tablet_manager.cc:1918] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:53.670656 32152 raft_consensus.cc:2229] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:24:53.666311 32153 ts_tablet_manager.cc:1905] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:53.670857 32151 log.cc:1196] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/d69e5c6df9f7408d83e21d77e3e80aab
I20260926 02:24:53.671536 32151 ts_tablet_manager.cc:1939] T d69e5c6df9f7408d83e21d77e3e80aab P 23b1b2e6ed0f416b8a29d9c2a5d56d71: Deleting consensus metadata
I20260926 02:24:53.671624 32152 raft_consensus.cc:2258] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:53.674515 31636 catalog_manager.cc:4841] TS 23b1b2e6ed0f416b8a29d9c2a5d56d71 (127.12.244.194:37819): tablet d69e5c6df9f7408d83e21d77e3e80aab (table test-table [id=830e6afb5ead4b879c277912c656c80a]) successfully deleted
I20260926 02:24:53.674996 32152 ts_tablet_manager.cc:1905] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:24:53 dist-test-slave-qfmw krb5kdc[31604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:24:53.684183 32152 ts_tablet_manager.cc:1918] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:53.684388 32152 log.cc:1196] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/d69e5c6df9f7408d83e21d77e3e80aab
I20260926 02:24:53.684482 32153 ts_tablet_manager.cc:1918] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:53.684682 32153 log.cc:1196] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/d69e5c6df9f7408d83e21d77e3e80aab
I20260926 02:24:53.685038 32152 ts_tablet_manager.cc:1939] T d69e5c6df9f7408d83e21d77e3e80aab P ab7009655c3443efa6333ae6f2152de6: Deleting consensus metadata
I20260926 02:24:53.685268 32153 ts_tablet_manager.cc:1939] T d69e5c6df9f7408d83e21d77e3e80aab P 66410791f1cc478eba8273375a3275ad: Deleting consensus metadata
I20260926 02:24:53.686983 31634 catalog_manager.cc:4841] TS ab7009655c3443efa6333ae6f2152de6 (127.12.244.195:44439): tablet d69e5c6df9f7408d83e21d77e3e80aab (table test-table [id=830e6afb5ead4b879c277912c656c80a]) successfully deleted
I20260926 02:24:53.687552 31634 catalog_manager.cc:4841] TS 66410791f1cc478eba8273375a3275ad (127.12.244.193:45099): tablet d69e5c6df9f7408d83e21d77e3e80aab (table test-table [id=830e6afb5ead4b879c277912c656c80a]) successfully deleted
I20260926 02:24:53.688691 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.656318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.12.244.193:45099)
0926 02:24:53.656517 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.656524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.659700 (+ 3176us) server_negotiation.cc:407] Connection header received
0926 02:24:53.663300 (+ 3600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.663307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.663402 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.663513 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:53.666082 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.667379 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.669524 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.670180 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.670298 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.680115 (+ 9817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:53.680136 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:53.680160 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:53.680264 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:53.685062 (+ 4798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.686219 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.686231 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.686235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.686439 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.686932 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.686939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.686945 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.687526 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:53.687875 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.688123 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.688292 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:24:53.689357 31835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:53.797304 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31711
I20260926 02:24:53.815798 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31846
Sep 26 02:24:53 dist-test-slave-qfmw krb5kdc[31604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:24:53.834681 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.816741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.12.244.194:37819)
0926 02:24:53.816934 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.816940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.816963 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:24:53.817150 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.817156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.817241 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.817373 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:53.818718 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.819917 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.821231 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.821947 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.822083 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.827016 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:53.827040 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:53.827063 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:53.827134 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:53.831008 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.831871 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.831884 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.831889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.832053 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.832421 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.832433 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.832437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.832993 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:53.833294 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.833627 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.833864 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:24:53.835537 31970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:53.926725 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31846
I20260926 02:24:53.945315 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31981
Sep 26 02:24:53 dist-test-slave-qfmw krb5kdc[31604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:24:53.964947 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:53.946227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.12.244.195:44439)
0926 02:24:53.946427 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:53.946438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:53.946658 (+ 220us) server_negotiation.cc:407] Connection header received
0926 02:24:53.946812 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:53.946818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:53.946902 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:53.947042 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:53.948496 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.949760 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.951352 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:53.952084 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:53.952247 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:53.957075 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:53.957090 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:53.957110 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:53.957162 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:53.961677 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.962546 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.962557 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.962561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.962750 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:53.963194 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:53.963200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:53.963203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:53.963729 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:53.964116 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:53.964306 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 02:24:53.964529 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:24:53.965801 32104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:54.090456 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31981
I20260926 02:24:54.111721 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 31617
I20260926 02:24:54.112893 31680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:54.226419 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 31617
2026-09-26T02:24:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4534 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T02:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:24:54Z Disabled control of system clock
I20260926 02:24:54.307344 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:37757
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:37757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:24:54.653952 32162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:54.654280 32162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:54.654366 32162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:54.661882 32162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:24:54.662003 32162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:54.662046 32162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:54.662097 32162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:24:54.662143 32162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:24:54.672533 32162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:37757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:37757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:54.673864 32162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:54.675813 32162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:54.689421 32169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:54.689514 32171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:54.689543 32168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:54.690851 32162 server_base.cc:1034] running on GCE node
I20260926 02:24:54.691824 32162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:54.693439 32162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:54.694694 32162 hybrid_clock.cc:648] HybridClock initialized: now 1790389494694621 us; error 49 us; skew 500 ppm
I20260926 02:24:54.695240 32162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:54.698810 32162 webserver.cc:460] Webserver started at http://127.12.244.254:32971/ using document root <none> and password file <none>
I20260926 02:24:54.699934 32162 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:54.700068 32162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:54.700475 32162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:54.704833 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "ed73f6c994574c87a7a5fc49705254ec"
format_stamp: "Formatted at 2026-09-26 02:24:54 on dist-test-slave-qfmw"
server_key: "65eefa8035268c5c4f070fe0839704f8"
server_key_iv: "884899b2355ec5c2f2776a0b285cb374"
server_key_version: "encryptionkey@0"
I20260926 02:24:54.705731 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "ed73f6c994574c87a7a5fc49705254ec"
format_stamp: "Formatted at 2026-09-26 02:24:54 on dist-test-slave-qfmw"
server_key: "65eefa8035268c5c4f070fe0839704f8"
server_key_iv: "884899b2355ec5c2f2776a0b285cb374"
server_key_version: "encryptionkey@0"
I20260926 02:24:54.712039 32162 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:24:54.716106 32177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:54.717784 32162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260926 02:24:54.718034 32162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "ed73f6c994574c87a7a5fc49705254ec"
format_stamp: "Formatted at 2026-09-26 02:24:54 on dist-test-slave-qfmw"
server_key: "65eefa8035268c5c4f070fe0839704f8"
server_key_iv: "884899b2355ec5c2f2776a0b285cb374"
server_key_version: "encryptionkey@0"
I20260926 02:24:54.718294 32162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:54.735932 32162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:54.736876 32162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:54.737190 32162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:54.756541 32162 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:37757
I20260926 02:24:54.756563 32228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:37757 every 8 connection(s)
I20260926 02:24:54.758080 32162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:24:54.762112 32229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:54.763067 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 32162
I20260926 02:24:54.763301 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:24:54.763789 13267 external_mini_cluster.cc:1428] Setting key 4fc4d0aa1f0ca676652d25caa9bd2ed2
I20260926 02:24:54.775477 32229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec: Bootstrap starting.
I20260926 02:24:54.777760 32232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:54.765778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.12.244.254:37757)
0926 02:24:54.766862 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:54.766877 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:54.766925 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:24:54.768759 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:54.768783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:54.769309 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:54.769714 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:54.770296 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:54.771659 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:54.773053 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:54.773765 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:54.774009 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:54.775284 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:54.775292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:54.775308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:54.775951 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:54.776039 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:54.776061 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:54.776199 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:54.776423 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 02:24:54.776767 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":431,"threads_started":1}
I20260926 02:24:54.779243 32229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:54.780383 32229 log.cc:824] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:54.783331 32229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec: No bootstrap required, opened a new log
I20260926 02:24:54.791770 32229 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed73f6c994574c87a7a5fc49705254ec" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37757 } }
I20260926 02:24:54.792228 32229 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:54.792346 32229 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed73f6c994574c87a7a5fc49705254ec, State: Initialized, Role: FOLLOWER
I20260926 02:24:54.792946 32229 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed73f6c994574c87a7a5fc49705254ec" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37757 } }
I20260926 02:24:54.793215 32229 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:24:54.793363 32229 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:24:54.793550 32229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:54.796592 32229 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed73f6c994574c87a7a5fc49705254ec" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37757 } }
I20260926 02:24:54.797283 32229 leader_election.cc:302] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed73f6c994574c87a7a5fc49705254ec; no voters:
I20260926 02:24:54.797923 32229 leader_election.cc:288] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:24:54.798111 32234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:54.798934 32234 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [term 1 LEADER]: Becoming Leader. State: Replica: ed73f6c994574c87a7a5fc49705254ec, State: Running, Role: LEADER
I20260926 02:24:54.799489 32234 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed73f6c994574c87a7a5fc49705254ec" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37757 } }
I20260926 02:24:54.800316 32229 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:24:54.802727 32236 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed73f6c994574c87a7a5fc49705254ec. Latest consensus state: current_term: 1 leader_uuid: "ed73f6c994574c87a7a5fc49705254ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed73f6c994574c87a7a5fc49705254ec" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37757 } } }
I20260926 02:24:54.802572 32235 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed73f6c994574c87a7a5fc49705254ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed73f6c994574c87a7a5fc49705254ec" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 37757 } } }
I20260926 02:24:54.803082 32236 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:54.803234 32235 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:54.804885 32242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:24:54.809763 32242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:24:54.819604 32242 catalog_manager.cc:1347] Generated new cluster ID: fd4ab26cbbd64e87bb36e9c4c5e752ed
I20260926 02:24:54.819800 32242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:24:54.842674 32242 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:24:54.844135 32242 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:24:54.855397 32242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec: Generated new TSK 0
I20260926 02:24:54.856395 32242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:24:54.866252 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37757
--builtin_ntp_servers=127.12.244.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:24:55.227892 32253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:55.228204 32253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:55.228292 32253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:55.235802 32253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:55.235924 32253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:55.236033 32253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:24:55.245918 32253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:55.247243 32253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:55.249070 32253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:55.263758 32259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:55.263780 32258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:55.263926 32261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:55.264971 32253 server_base.cc:1034] running on GCE node
I20260926 02:24:55.265841 32253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:55.267228 32253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:55.268537 32253 hybrid_clock.cc:648] HybridClock initialized: now 1790389495268501 us; error 45 us; skew 500 ppm
I20260926 02:24:55.268949 32253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:55.271994 32253 webserver.cc:460] Webserver started at http://127.12.244.193:37833/ using document root <none> and password file <none>
I20260926 02:24:55.272799 32253 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:55.272899 32253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:55.273283 32253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:55.276583 32253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "131783c1ec8b4231a60a29ab3274e78b"
format_stamp: "Formatted at 2026-09-26 02:24:55 on dist-test-slave-qfmw"
server_key: "77dc9f533818620ff841bd161d2dfcea"
server_key_iv: "79bf3d499aa1eeecebeb7848802483f8"
server_key_version: "encryptionkey@0"
I20260926 02:24:55.277483 32253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "131783c1ec8b4231a60a29ab3274e78b"
format_stamp: "Formatted at 2026-09-26 02:24:55 on dist-test-slave-qfmw"
server_key: "77dc9f533818620ff841bd161d2dfcea"
server_key_iv: "79bf3d499aa1eeecebeb7848802483f8"
server_key_version: "encryptionkey@0"
I20260926 02:24:55.283538 32253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 02:24:55.287837 32267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:55.289520 32253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:24:55.289757 32253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "131783c1ec8b4231a60a29ab3274e78b"
format_stamp: "Formatted at 2026-09-26 02:24:55 on dist-test-slave-qfmw"
server_key: "77dc9f533818620ff841bd161d2dfcea"
server_key_iv: "79bf3d499aa1eeecebeb7848802483f8"
server_key_version: "encryptionkey@0"
I20260926 02:24:55.290019 32253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:55.307951 32253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:55.308888 32253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:55.309233 32253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:55.310295 32253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:55.312170 32253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:55.312312 32253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:55.312451 32253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:55.312556 32253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:55.347544 32253 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:39143
I20260926 02:24:55.347609 32380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:39143 every 8 connection(s)
I20260926 02:24:55.349012 32253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:24:55.353772 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 32253
I20260926 02:24:55.353937 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:24:55.354400 13267 external_mini_cluster.cc:1428] Setting key 5df6b57912324825d26b973c3707d6c0
I20260926 02:24:55.357620 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37757
--builtin_ntp_servers=127.12.244.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:24:55.364583 32383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:55.351750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:51103 (local address 127.12.244.254:37757)
0926 02:24:55.352243 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:55.352250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:55.353549 (+ 1299us) server_negotiation.cc:407] Connection header received
0926 02:24:55.356031 (+ 2482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:55.356035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:55.356146 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:55.356258 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:55.357590 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.358857 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.360687 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.361423 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.361581 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:55.363400 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:55.363409 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:55.363417 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:55.363446 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:55.363501 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:55.363521 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:55.363623 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:55.363965 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 02:24:55.364100 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260926 02:24:55.365670 32384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:55.352403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37757 (local address 127.12.244.193:51103)
0926 02:24:55.353320 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:55.353357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:55.355700 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:55.356615 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:55.356629 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:55.356751 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:55.357396 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:55.357416 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.359052 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.359056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:55.360510 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:55.360525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.361737 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.361744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:55.362064 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:55.362936 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:24:55.362939 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:55.362993 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:24:55.363155 (+ 162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:55.363667 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:55.363680 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:55.363691 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:24:55.363914 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 02:24:55.364317 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260926 02:24:55.367977 32381 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37757
I20260926 02:24:55.368391 32381 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:55.369280 32381 heartbeater.cc:511] Master 127.12.244.254:37757 requested a full tablet report, sending...
I20260926 02:24:55.372056 32194 ts_manager.cc:194] Registered new tserver with Master: 131783c1ec8b4231a60a29ab3274e78b (127.12.244.193:39143)
I20260926 02:24:55.374610 32194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.193:51103
W20260926 02:24:55.714603 32385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:55.714915 32385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:55.715009 32385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:55.722633 32385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:55.722757 32385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:55.722950 32385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:24:55.732791 32385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:55.734177 32385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:55.736133 32385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:55.750541 32390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:55.750499 32391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:55.750541 32393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:55.751287 32385 server_base.cc:1034] running on GCE node
I20260926 02:24:55.751995 32385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:55.753039 32385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:55.754254 32385 hybrid_clock.cc:648] HybridClock initialized: now 1790389495754201 us; error 52 us; skew 500 ppm
I20260926 02:24:55.754581 32385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:55.761581 32385 webserver.cc:460] Webserver started at http://127.12.244.194:43597/ using document root <none> and password file <none>
I20260926 02:24:55.762554 32385 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:55.762688 32385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:55.763022 32385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:55.767061 32385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "b2a3f46110204d4886b4f74a92297524"
format_stamp: "Formatted at 2026-09-26 02:24:55 on dist-test-slave-qfmw"
server_key: "fdb753cde1cc709992d3c6f328f112a9"
server_key_iv: "b042b7203ee41f3758f2e08186061998"
server_key_version: "encryptionkey@0"
I20260926 02:24:55.768276 32385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "b2a3f46110204d4886b4f74a92297524"
format_stamp: "Formatted at 2026-09-26 02:24:55 on dist-test-slave-qfmw"
server_key: "fdb753cde1cc709992d3c6f328f112a9"
server_key_iv: "b042b7203ee41f3758f2e08186061998"
server_key_version: "encryptionkey@0"
I20260926 02:24:55.775424 32385 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 02:24:55.779700 32399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:55.781278 32385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:24:55.781438 32385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "b2a3f46110204d4886b4f74a92297524"
format_stamp: "Formatted at 2026-09-26 02:24:55 on dist-test-slave-qfmw"
server_key: "fdb753cde1cc709992d3c6f328f112a9"
server_key_iv: "b042b7203ee41f3758f2e08186061998"
server_key_version: "encryptionkey@0"
I20260926 02:24:55.781648 32385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:55.799933 32385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:55.800873 32385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:55.801139 32385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:55.802217 32385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:55.803794 32385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:55.803882 32385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:55.803958 32385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:55.804013 32385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:55.838272 32385 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:46613
I20260926 02:24:55.838325 32512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:46613 every 8 connection(s)
I20260926 02:24:55.839660 32385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:24:55.849056 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 32385
I20260926 02:24:55.849273 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:24:55.849795 13267 external_mini_cluster.cc:1428] Setting key d79d79e7cbe65ab3b8f9ecd902db3883
I20260926 02:24:55.853010 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37757
--builtin_ntp_servers=127.12.244.212:39699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:24:55.853235 32383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:55.842338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:42383 (local address 127.12.244.254:37757)
0926 02:24:55.842524 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:55.842528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:55.844025 (+ 1497us) server_negotiation.cc:407] Connection header received
0926 02:24:55.845904 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:55.845910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:55.846007 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:55.846132 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:55.847323 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.848118 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.849979 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.850539 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.850720 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:55.852333 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:55.852340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:55.852345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:55.852368 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:55.852406 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:55.852433 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:55.852527 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:55.852766 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 02:24:55.852883 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:24:55.854295 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:55.842884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37757 (local address 127.12.244.194:42383)
0926 02:24:55.843854 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:55.843900 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:55.845610 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:55.846392 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:55.846401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:55.846502 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:55.847152 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:55.847172 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.848281 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.848287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:55.849777 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:55.849790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:55.850826 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:55.850833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:55.851149 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:55.851949 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:24:55.851952 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:55.851998 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:24:55.852130 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:55.852538 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:55.852545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:55.852551 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:24:55.852696 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 02:24:55.853043 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":323,"threads_started":1}
I20260926 02:24:55.855877 32513 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37757
I20260926 02:24:55.856217 32513 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:55.857028 32513 heartbeater.cc:511] Master 127.12.244.254:37757 requested a full tablet report, sending...
I20260926 02:24:55.858693 32194 ts_manager.cc:194] Registered new tserver with Master: b2a3f46110204d4886b4f74a92297524 (127.12.244.194:46613)
I20260926 02:24:55.859975 32194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.194:42383
W20260926 02:24:56.204408 32516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:56.204725 32516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:56.204869 32516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:56.212740 32516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:56.212865 32516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:56.212975 32516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:24:56.222657 32516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:39699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:37757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:56.224138 32516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:56.226024 32516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:56.240058 32522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:56.240538 32524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:56.240610 32521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:56.241222 32516 server_base.cc:1034] running on GCE node
I20260926 02:24:56.241890 32516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:56.243062 32516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:56.244366 32516 hybrid_clock.cc:648] HybridClock initialized: now 1790389496244286 us; error 62 us; skew 500 ppm
I20260926 02:24:56.244745 32516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:24:56.247363 32516 webserver.cc:460] Webserver started at http://127.12.244.195:42631/ using document root <none> and password file <none>
I20260926 02:24:56.248128 32516 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:56.248253 32516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:56.248637 32516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:56.251879 32516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "4f2a8cfe772545189772e8d34562337c"
format_stamp: "Formatted at 2026-09-26 02:24:56 on dist-test-slave-qfmw"
server_key: "7513e34a8a24c5a02a63c127d706bf3e"
server_key_iv: "e4e4f5efc4b8539542eca8d749e95689"
server_key_version: "encryptionkey@0"
I20260926 02:24:56.252755 32516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "4f2a8cfe772545189772e8d34562337c"
format_stamp: "Formatted at 2026-09-26 02:24:56 on dist-test-slave-qfmw"
server_key: "7513e34a8a24c5a02a63c127d706bf3e"
server_key_iv: "e4e4f5efc4b8539542eca8d749e95689"
server_key_version: "encryptionkey@0"
I20260926 02:24:56.258267 32516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:24:56.262159 32530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:56.263772 32516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:24:56.263931 32516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "4f2a8cfe772545189772e8d34562337c"
format_stamp: "Formatted at 2026-09-26 02:24:56 on dist-test-slave-qfmw"
server_key: "7513e34a8a24c5a02a63c127d706bf3e"
server_key_iv: "e4e4f5efc4b8539542eca8d749e95689"
server_key_version: "encryptionkey@0"
I20260926 02:24:56.264134 32516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:56.279717 32516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:56.280582 32516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:56.280859 32516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:56.281811 32516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:56.283339 32516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:56.283439 32516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:56.283524 32516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:56.283577 32516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:56.318975 32516 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:41913
I20260926 02:24:56.319027 32644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:41913 every 8 connection(s)
I20260926 02:24:56.320406 32516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:24:56.330464 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 32516
I20260926 02:24:56.330643 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:24:56.331208 13267 external_mini_cluster.cc:1428] Setting key 5f39c960a00eef8a0049eb0dfd2c9514
I20260926 02:24:56.334782 32383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.323252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:40525 (local address 127.12.244.254:37757)
0926 02:24:56.323439 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.323444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.324881 (+ 1437us) server_negotiation.cc:407] Connection header received
0926 02:24:56.326801 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.326806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.326882 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.326977 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:56.328480 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.329270 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.331251 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.331790 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.331932 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.333837 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:56.333843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:56.333847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:56.333869 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:56.333907 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:56.333924 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:56.334033 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.334328 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.334443 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:24:56.335994 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.323781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:37757 (local address 127.12.244.195:40525)
0926 02:24:56.324715 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.324753 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:56.326490 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:56.327288 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:56.327297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:56.327407 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:56.328257 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.328275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.329464 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.329514 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.331007 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.331021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.332153 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.332162 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.332519 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.333441 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:24:56.333444 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:56.333507 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:24:56.333622 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:56.334050 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:56.334061 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:56.334068 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:24:56.334286 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 02:24:56.334717 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":273,"threads_started":1}
I20260926 02:24:56.337616 32645 heartbeater.cc:347] Connected to a master server at 127.12.244.254:37757
I20260926 02:24:56.337913 32645 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:56.338687 32645 heartbeater.cc:511] Master 127.12.244.254:37757 requested a full tablet report, sending...
I20260926 02:24:56.340286 32194 ts_manager.cc:194] Registered new tserver with Master: 4f2a8cfe772545189772e8d34562337c (127.12.244.195:41913)
I20260926 02:24:56.341149 32194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.195:40525
I20260926 02:24:56.347689 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:24:56.360201 32383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.353361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.12.244.254:37757)
0926 02:24:56.353586 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.353595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.354067 (+ 472us) server_negotiation.cc:407] Connection header received
0926 02:24:56.354276 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.354281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.354351 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.354445 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:56.355045 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.355892 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.357441 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.357888 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.358023 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.359269 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:56.359274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:56.359277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:56.359297 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:56.359348 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:56.359364 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:56.359571 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.359775 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.359860 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:24:56.364533 32194 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:24:56.371920 32194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:24:56.378398 32381 heartbeater.cc:503] Master 127.12.244.254:37757 was elected leader, sending a full tablet report...
I20260926 02:24:56.408272 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.386180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.12.244.195:41913)
0926 02:24:56.397441 (+ 11261us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.397455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.398387 (+ 932us) server_negotiation.cc:407] Connection header received
0926 02:24:56.398598 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.398612 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.399140 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.399424 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:56.400889 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.402581 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.405205 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.406834 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.407052 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.407333 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.407716 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.407893 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10984,"thread_start_us":11277,"threads_started":1}
I20260926 02:24:56.408800 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.385965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:41913 (local address 127.0.0.1:52766)
0926 02:24:56.398208 (+ 12243us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.398234 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:56.398397 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:56.399858 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:56.399871 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:56.399912 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:56.400640 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.400657 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.402752 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.402763 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.404973 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.404986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.407052 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.407058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.407375 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.407461 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 02:24:56.407599 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 02:24:56.407812 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12071,"mutex_wait_us":97,"thread_start_us":11921,"threads_started":1}
I20260926 02:24:56.409955 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.385654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52658)
0926 02:24:56.386668 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.386730 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:56.387049 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:56.400064 (+ 13015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:56.400073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:56.400120 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:56.400734 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.400755 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.402555 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.402561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.405944 (+ 3383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.405961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.408328 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.408334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.409309 (+ 975us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.409347 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 02:24:56.409424 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 02:24:56.409562 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":684,"thread_start_us":995,"threads_started":1}
I20260926 02:24:56.410575 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.388292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:46613 (local address 127.0.0.1:54782)
0926 02:24:56.400805 (+ 12513us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.400832 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:56.400941 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:56.402664 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:56.402670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:56.402707 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:56.403322 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.403342 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.405371 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.405377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.407427 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.407442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.409730 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.409736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.409952 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.409994 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:24:56.410074 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:24:56.410217 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12345,"mutex_wait_us":26,"thread_start_us":12713,"threads_started":1}
I20260926 02:24:56.410835 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.385909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.12.244.193:39143)
0926 02:24:56.399019 (+ 13110us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.399029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.399064 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:24:56.399189 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.399198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.399464 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.399698 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:56.401019 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.402381 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.406184 (+ 3803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.407780 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.407987 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.408196 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.410232 (+ 2036us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.410428 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12811,"thread_start_us":13394,"threads_started":1}
I20260926 02:24:56.411397 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.388776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.12.244.194:46613)
0926 02:24:56.401758 (+ 12982us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.401768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.401798 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:24:56.401895 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.401904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.402160 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.402390 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:56.403536 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.405137 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.407652 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.409228 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.409382 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.409600 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.410928 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.411134 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12814,"thread_start_us":12679,"threads_started":1}
I20260926 02:24:56.413355 32579 tablet_service.cc:1467] Processing CreateTablet for tablet 4d52e70ab9f74897a217d7042472d20c (DEFAULT_TABLE table=test-table [id=e8809e014aa347b4bf33a111e0f8f942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:56.414407 32315 tablet_service.cc:1467] Processing CreateTablet for tablet 4d52e70ab9f74897a217d7042472d20c (DEFAULT_TABLE table=test-table [id=e8809e014aa347b4bf33a111e0f8f942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:56.415071 32579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d52e70ab9f74897a217d7042472d20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:56.415084 32448 tablet_service.cc:1467] Processing CreateTablet for tablet 4d52e70ab9f74897a217d7042472d20c (DEFAULT_TABLE table=test-table [id=e8809e014aa347b4bf33a111e0f8f942]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:24:56.415938 32315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d52e70ab9f74897a217d7042472d20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:56.416440 32448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d52e70ab9f74897a217d7042472d20c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:56.433243 32663 tablet_bootstrap.cc:492] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: Bootstrap starting.
I20260926 02:24:56.434574 32665 tablet_bootstrap.cc:492] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Bootstrap starting.
I20260926 02:24:56.436779 32663 tablet_bootstrap.cc:654] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:56.437903 32663 log.cc:824] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:56.439670 32665 tablet_bootstrap.cc:654] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:56.441399 32663 tablet_bootstrap.cc:492] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: No bootstrap required, opened a new log
I20260926 02:24:56.441354 32665 log.cc:824] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:56.441744 32663 ts_tablet_manager.cc:1397] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260926 02:24:56.442449 32664 tablet_bootstrap.cc:492] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Bootstrap starting.
I20260926 02:24:56.446427 32664 tablet_bootstrap.cc:654] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:56.447850 32664 log.cc:824] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:24:56.450212 32665 tablet_bootstrap.cc:492] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: No bootstrap required, opened a new log
I20260926 02:24:56.450516 32665 ts_tablet_manager.cc:1397] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260926 02:24:56.450991 32664 tablet_bootstrap.cc:492] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: No bootstrap required, opened a new log
I20260926 02:24:56.451287 32664 ts_tablet_manager.cc:1397] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 02:24:56.452453 32663 raft_consensus.cc:348] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.452941 32663 raft_consensus.cc:374] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:56.453055 32663 raft_consensus.cc:729] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131783c1ec8b4231a60a29ab3274e78b, State: Initialized, Role: FOLLOWER
I20260926 02:24:56.453717 32663 consensus_queue.cc:260] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.456962 32663 ts_tablet_manager.cc:1428] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 02:24:56.460060 32665 raft_consensus.cc:348] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.460582 32665 raft_consensus.cc:374] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:56.460716 32665 raft_consensus.cc:729] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a3f46110204d4886b4f74a92297524, State: Initialized, Role: FOLLOWER
I20260926 02:24:56.461360 32665 consensus_queue.cc:260] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.461310 32664 raft_consensus.cc:348] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.461804 32664 raft_consensus.cc:374] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:56.461912 32664 raft_consensus.cc:729] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2a8cfe772545189772e8d34562337c, State: Initialized, Role: FOLLOWER
I20260926 02:24:56.462680 32513 heartbeater.cc:503] Master 127.12.244.254:37757 was elected leader, sending a full tablet report...
I20260926 02:24:56.462574 32664 consensus_queue.cc:260] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.463416 32665 ts_tablet_manager.cc:1428] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260926 02:24:56.463800 32645 heartbeater.cc:503] Master 127.12.244.254:37757 was elected leader, sending a full tablet report...
I20260926 02:24:56.465651 32664 ts_tablet_manager.cc:1428] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260926 02:24:56.572261 32646 tablet.cc:2369] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:56.591717 32514 tablet.cc:2369] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:24:56.601827 32382 tablet.cc:2369] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:24:56.651371 32671 raft_consensus.cc:482] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:24:56.651671 32671 raft_consensus.cc:504] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.653429 32671 leader_election.cc:288] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 131783c1ec8b4231a60a29ab3274e78b (127.12.244.193:39143), b2a3f46110204d4886b4f74a92297524 (127.12.244.194:46613)
I20260926 02:24:56.660236 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.653901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:46613 (local address 127.12.244.195:46953)
0926 02:24:56.654104 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.654133 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:56.654263 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:56.654654 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:56.654659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:56.654687 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:56.654886 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.654898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.656190 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.656193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.658097 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.658112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.659472 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.659482 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.659688 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.659731 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 02:24:56.659826 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 02:24:56.659930 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":30}
I20260926 02:24:56.660306 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.654051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:46953 (local address 127.12.244.194:46613)
0926 02:24:56.654261 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.654267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.654292 (+ 25us) server_negotiation.cc:407] Connection header received
0926 02:24:56.654399 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.654403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.654474 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.654586 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:56.655066 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.656015 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.658279 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.659281 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.659446 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.659516 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.659880 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.660021 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:24:56.660908 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.654057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:58959 (local address 127.12.244.193:39143)
0926 02:24:56.654267 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.654273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.655313 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 02:24:56.655423 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.655428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.655517 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.655609 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:24:56.656127 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.656810 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.658911 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.660070 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.660238 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.660301 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.660650 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.660747 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:24:56.661427 32672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.653944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.12.244.195:58959)
0926 02:24:56.655211 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.655226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:56.655322 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:56.655633 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:56.655636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:56.655654 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:24:56.655958 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.655966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.656983 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.656988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.658721 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:56.658731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.660246 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.660255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:56.660448 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.660488 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 02:24:56.661098 (+ 610us) client_negotiation.cc:240] Negotiation successful
0926 02:24:56.661202 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"mutex_wait_us":389,"thread_start_us":475,"threads_started":1}
I20260926 02:24:56.661314 32468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d52e70ab9f74897a217d7042472d20c" candidate_uuid: "4f2a8cfe772545189772e8d34562337c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a3f46110204d4886b4f74a92297524" is_pre_election: true
I20260926 02:24:56.661872 32468 raft_consensus.cc:2454] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2a8cfe772545189772e8d34562337c in term 0.
I20260926 02:24:56.662456 32336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d52e70ab9f74897a217d7042472d20c" candidate_uuid: "4f2a8cfe772545189772e8d34562337c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131783c1ec8b4231a60a29ab3274e78b" is_pre_election: true
I20260926 02:24:56.662573 32533 leader_election.cc:302] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2a8cfe772545189772e8d34562337c, b2a3f46110204d4886b4f74a92297524; no voters:
I20260926 02:24:56.663005 32336 raft_consensus.cc:2454] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2a8cfe772545189772e8d34562337c in term 0.
I20260926 02:24:56.663005 32671 raft_consensus.cc:2781] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:24:56.663235 32671 raft_consensus.cc:482] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:24:56.663353 32671 raft_consensus.cc:3037] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:56.665624 32671 raft_consensus.cc:504] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.666306 32671 leader_election.cc:288] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [CANDIDATE]: Term 1 election: Requested vote from peers 131783c1ec8b4231a60a29ab3274e78b (127.12.244.193:39143), b2a3f46110204d4886b4f74a92297524 (127.12.244.194:46613)
I20260926 02:24:56.666788 32336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d52e70ab9f74897a217d7042472d20c" candidate_uuid: "4f2a8cfe772545189772e8d34562337c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "131783c1ec8b4231a60a29ab3274e78b"
I20260926 02:24:56.666860 32468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d52e70ab9f74897a217d7042472d20c" candidate_uuid: "4f2a8cfe772545189772e8d34562337c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a3f46110204d4886b4f74a92297524"
I20260926 02:24:56.667011 32336 raft_consensus.cc:3037] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:56.667049 32468 raft_consensus.cc:3037] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:56.670347 32336 raft_consensus.cc:2454] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2a8cfe772545189772e8d34562337c in term 1.
I20260926 02:24:56.670349 32468 raft_consensus.cc:2454] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2a8cfe772545189772e8d34562337c in term 1.
I20260926 02:24:56.670823 32533 leader_election.cc:302] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2a8cfe772545189772e8d34562337c, b2a3f46110204d4886b4f74a92297524; no voters:
I20260926 02:24:56.671245 32671 raft_consensus.cc:2781] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:56.671674 32671 raft_consensus.cc:686] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 1 LEADER]: Becoming Leader. State: Replica: 4f2a8cfe772545189772e8d34562337c, State: Running, Role: LEADER
I20260926 02:24:56.672202 32671 consensus_queue.cc:237] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } }
I20260926 02:24:56.677121 32194 catalog_manager.cc:5495] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f2a8cfe772545189772e8d34562337c (127.12.244.195). New cstate: current_term: 1 leader_uuid: "4f2a8cfe772545189772e8d34562337c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f2a8cfe772545189772e8d34562337c" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 41913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 } health_report { overall_health: UNKNOWN } } }
I20260926 02:24:56.707541 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.700750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.12.244.195:41913)
0926 02:24:56.700971 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.700977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.700998 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:24:56.701120 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.701126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.701235 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.701395 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:24:56.701841 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.703004 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.705253 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.705690 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.705798 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.706357 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:24:56.706985 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:24:56.707183 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.707302 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.707342 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:24:56.717288 32336 raft_consensus.cc:1261] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 1 FOLLOWER]: Refusing update from remote peer 4f2a8cfe772545189772e8d34562337c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:56.717288 32468 raft_consensus.cc:1261] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 1 FOLLOWER]: Refusing update from remote peer 4f2a8cfe772545189772e8d34562337c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:24:56.718412 32673 consensus_queue.cc:1035] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a3f46110204d4886b4f74a92297524" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 46613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:56.718863 32671 consensus_queue.cc:1035] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [LEADER]: Connected to new peer: Peer: permanent_uuid: "131783c1ec8b4231a60a29ab3274e78b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 39143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:24:56.746412 32194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:56.746685 32194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:24:56.750734 32194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed73f6c994574c87a7a5fc49705254ec: Sending DeleteTablet for 3 replicas of tablet 4d52e70ab9f74897a217d7042472d20c
I20260926 02:24:56.752110 32315 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d52e70ab9f74897a217d7042472d20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:56 UTC) from {username='slave'} at 127.0.0.1:52658
I20260926 02:24:56.752476 32579 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d52e70ab9f74897a217d7042472d20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:56 UTC) from {username='slave'} at 127.0.0.1:52766
I20260926 02:24:56.752682 32448 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d52e70ab9f74897a217d7042472d20c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:24:56 UTC) from {username='slave'} at 127.0.0.1:54782
I20260926 02:24:56.754995 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 32253
I20260926 02:24:56.766098 32687 tablet_replica.cc:330] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: stopping tablet replica
I20260926 02:24:56.766897 32686 tablet_replica.cc:330] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: stopping tablet replica
I20260926 02:24:56.768172 32686 raft_consensus.cc:2229] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:24:56.768893 32687 raft_consensus.cc:2229] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:56.770316 32687 raft_consensus.cc:2258] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:56.771962 32686 raft_consensus.cc:2258] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:56.776106 32687 ts_tablet_manager.cc:1905] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:56.779325 32686 ts_tablet_manager.cc:1905] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:56.785408 32685 tablet_replica.cc:330] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: stopping tablet replica
I20260926 02:24:56.786085 32685 raft_consensus.cc:2229] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:24:56.786648 32685 raft_consensus.cc:2258] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:24:56.786742 32686 ts_tablet_manager.cc:1918] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:56.786926 32686 log.cc:1196] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/4d52e70ab9f74897a217d7042472d20c
I20260926 02:24:56.786891 32687 ts_tablet_manager.cc:1918] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:24:56.787212 32687 log.cc:1196] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/4d52e70ab9f74897a217d7042472d20c
I20260926 02:24:56.788318 32687 ts_tablet_manager.cc:1939] T 4d52e70ab9f74897a217d7042472d20c P b2a3f46110204d4886b4f74a92297524: Deleting consensus metadata
I20260926 02:24:56.788967 32685 ts_tablet_manager.cc:1905] T 4d52e70ab9f74897a217d7042472d20c P 131783c1ec8b4231a60a29ab3274e78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:24:56.789068 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.759959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.12.244.193:39143)
0926 02:24:56.760222 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.760230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.777213 (+ 16983us) server_negotiation.cc:407] Connection header received
0926 02:24:56.777490 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.777497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.777641 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.777718 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:56.778355 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.779514 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.783553 (+ 4039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.784252 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.784355 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.787328 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:56.787340 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:56.787357 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:56.788162 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:56.788224 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:56.788256 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:56.788337 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.788590 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.788720 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 02:24:56.790169 32376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:56.790642 32180 catalog_manager.cc:4841] TS b2a3f46110204d4886b4f74a92297524 (127.12.244.194:46613): tablet 4d52e70ab9f74897a217d7042472d20c (table test-table [id=e8809e014aa347b4bf33a111e0f8f942]) successfully deleted
I20260926 02:24:56.791318 32686 ts_tablet_manager.cc:1939] T 4d52e70ab9f74897a217d7042472d20c P 4f2a8cfe772545189772e8d34562337c: Deleting consensus metadata
I20260926 02:24:56.793072 32179 catalog_manager.cc:4841] TS 4f2a8cfe772545189772e8d34562337c (127.12.244.195:41913): tablet 4d52e70ab9f74897a217d7042472d20c (table test-table [id=e8809e014aa347b4bf33a111e0f8f942]) successfully deleted
I20260926 02:24:56.899209 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 32253
W20260926 02:24:56.917125 32178 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.244.193:39143 (error 108)
I20260926 02:24:56.917850 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 32385
I20260926 02:24:56.919693 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.918448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52672)
0926 02:24:56.919342 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.919480 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":530,"threads_started":1}
W20260926 02:24:56.919996 32178 catalog_manager.cc:4584] TS 131783c1ec8b4231a60a29ab3274e78b (127.12.244.193:39143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d52e70ab9f74897a217d7042472d20c: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
I20260926 02:24:56.925753 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.918666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.12.244.194:46613)
0926 02:24:56.918855 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:56.918862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:56.919254 (+ 392us) server_negotiation.cc:407] Connection header received
0926 02:24:56.919321 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:56.919324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:56.919416 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:56.919536 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:56.919991 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.920884 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.922302 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:56.922795 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:56.923011 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:56.924338 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:56.924347 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:56.924362 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:56.924949 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:56.925008 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:56.925033 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:56.925158 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:56.925303 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 02:24:56.925415 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:24:56.926450 32508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:56.973670 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.973198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52674)
0926 02:24:56.973418 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.973522 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 02:24:56.974934 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:56.974507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52676)
0926 02:24:56.974709 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:56.974783 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 02:24:57.035952 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:57.035340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52682)
0926 02:24:57.035630 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:57.035755 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 02:24:57.037284 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:57.036814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52684)
0926 02:24:57.036997 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:57.037089 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 02:24:57.066219 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 32385
I20260926 02:24:57.086410 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 32516
I20260926 02:24:57.095011 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:57.087329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.12.244.195:41913)
0926 02:24:57.087504 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:57.087508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:57.087532 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:24:57.087780 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:57.087787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:57.087886 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:57.087975 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:57.088456 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:57.089370 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:57.091291 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:57.091781 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:57.091881 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:57.093279 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:57.093288 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:57.093304 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:24:57.094023 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:57.094079 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:24:57.094099 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:57.094190 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:57.094448 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 02:24:57.094544 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:24:57.095858 32640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:57.118700 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:57.118198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52692)
0926 02:24:57.118438 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:57.118538 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 02:24:57.120023 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:57.119537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52708)
0926 02:24:57.119736 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:57.119854 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 02:24:57.221282 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 32516
I20260926 02:24:57.241276 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 32162
I20260926 02:24:57.242558 32224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:24:57.353425 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:57.352832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:39143 (local address 127.0.0.1:52710)
0926 02:24:57.353060 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:57.353224 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.193:39143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 02:24:57.354094 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 32162
2026-09-26T02:24:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3127 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:24:57 dist-test-slave-qfmw krb5kdc[32697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:24:57 dist-test-slave-qfmw krb5kdc[32697](info): set up 2 sockets
Sep 26 02:24:57 dist-test-slave-qfmw krb5kdc[32697](info): commencing operation
krb5kdc: starting...
W20260926 02:24:58.473811 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:24:58 dist-test-slave-qfmw krb5kdc[32697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:24:58.738788 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39945
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:59.090965 32710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:59.091434 32710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:59.091591 32710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:59.100502 32710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:24:59.100667 32710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:59.100783 32710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:59.100896 32710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:24:59.100979 32710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:24:59.111775 32710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:59.113374 32710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:59.114768 32710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:59.127890 32716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:59.128067 32715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:59.128070 32718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:59.129129 32710 server_base.cc:1034] running on GCE node
I20260926 02:24:59.130123 32710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:59.131917 32710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:59.133201 32710 hybrid_clock.cc:648] HybridClock initialized: now 1790389499133150 us; error 61 us; skew 500 ppm
I20260926 02:24:59.133591 32710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:59 dist-test-slave-qfmw krb5kdc[32697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389499, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:59.137388 32710 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:24:59.139379 32710 webserver.cc:460] Webserver started at http://127.12.244.254:38099/ using document root <none> and password file <none>
I20260926 02:24:59.140202 32710 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:59.140364 32710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:59.140702 32710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:59.144243 32710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "4fb9dcb7e91546738c2045e7592b0134"
format_stamp: "Formatted at 2026-09-26 02:24:59 on dist-test-slave-qfmw"
server_key: "fe9e508e5117ee5a1e3c29698030efe5"
server_key_iv: "eb8a4d0a39f50a4e2263a9c48cadaa87"
server_key_version: "encryptionkey@0"
I20260926 02:24:59.145169 32710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "4fb9dcb7e91546738c2045e7592b0134"
format_stamp: "Formatted at 2026-09-26 02:24:59 on dist-test-slave-qfmw"
server_key: "fe9e508e5117ee5a1e3c29698030efe5"
server_key_iv: "eb8a4d0a39f50a4e2263a9c48cadaa87"
server_key_version: "encryptionkey@0"
I20260926 02:24:59.151556 32710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260926 02:24:59.156105 32725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:59.158018 32710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:24:59.158280 32710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "4fb9dcb7e91546738c2045e7592b0134"
format_stamp: "Formatted at 2026-09-26 02:24:59 on dist-test-slave-qfmw"
server_key: "fe9e508e5117ee5a1e3c29698030efe5"
server_key_iv: "eb8a4d0a39f50a4e2263a9c48cadaa87"
server_key_version: "encryptionkey@0"
I20260926 02:24:59.158567 32710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:59.179574 32710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:59.184540 32710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:59.184875 32710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:59.206153 32710 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:39945
I20260926 02:24:59.206203 308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:39945 every 8 connection(s)
I20260926 02:24:59.207821 32710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:24:59.211892 309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:24:59.215124 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 32710
I20260926 02:24:59.215329 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:24:59.215878 13267 external_mini_cluster.cc:1428] Setting key d4b47aa47b3dc47034160343aa1ac5cf
I20260926 02:24:59.225740 309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134: Bootstrap starting.
I20260926 02:24:59.230082 309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134: Neither blocks nor log segments found. Creating new log.
I20260926 02:24:59.231640 309 log.cc:824] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:24:59 dist-test-slave-qfmw krb5kdc[32697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:59.235445 309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134: No bootstrap required, opened a new log
I20260926 02:24:59.243259 312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:59.218007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.12.244.254:39945)
0926 02:24:59.218941 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:59.218955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:59.219000 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:24:59.220840 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:59.220867 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:59.221401 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:59.221972 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:59.223939 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:59.225566 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:59.227640 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:59.228413 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:59.228611 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:59.234304 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:59.234314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:59.234336 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:59.234390 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:59.238228 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:59.239157 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:59.239168 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:59.239185 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:59.239389 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:59.239886 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:59.239891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:59.239897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:59.240558 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:59.240929 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:59.241483 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 02:24:59.241994 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":494,"threads_started":1}
I20260926 02:24:59.243765 309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb9dcb7e91546738c2045e7592b0134" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39945 } }
I20260926 02:24:59.244233 309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:24:59.244356 309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb9dcb7e91546738c2045e7592b0134, State: Initialized, Role: FOLLOWER
I20260926 02:24:59.245036 309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb9dcb7e91546738c2045e7592b0134" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39945 } }
I20260926 02:24:59.245304 309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:24:59.245466 309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:24:59.245661 309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:24:59.248510 309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb9dcb7e91546738c2045e7592b0134" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39945 } }
I20260926 02:24:59.249250 309 leader_election.cc:302] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fb9dcb7e91546738c2045e7592b0134; no voters:
I20260926 02:24:59.249934 309 leader_election.cc:288] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:24:59.250257 314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:24:59.251394 314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [term 1 LEADER]: Becoming Leader. State: Replica: 4fb9dcb7e91546738c2045e7592b0134, State: Running, Role: LEADER
I20260926 02:24:59.252053 314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb9dcb7e91546738c2045e7592b0134" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39945 } }
I20260926 02:24:59.252380 309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:24:59.256197 315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fb9dcb7e91546738c2045e7592b0134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb9dcb7e91546738c2045e7592b0134" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39945 } } }
I20260926 02:24:59.256161 316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fb9dcb7e91546738c2045e7592b0134. Latest consensus state: current_term: 1 leader_uuid: "4fb9dcb7e91546738c2045e7592b0134" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fb9dcb7e91546738c2045e7592b0134" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39945 } } }
I20260926 02:24:59.256654 315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:59.256690 316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134 [sys.catalog]: This master's current role is: LEADER
I20260926 02:24:59.259402 321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:24:59.265076 321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:24:59.276067 321 catalog_manager.cc:1347] Generated new cluster ID: b863ae6bef8f4d18895104b63fbc4018
I20260926 02:24:59.276218 321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:24:59.307044 321 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:24:59.308677 321 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:24:59.319880 321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134: Generated new TSK 0
I20260926 02:24:59.320888 321 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.244.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
Entry for principal kudu/127.12.244.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab.
I20260926 02:24:59.400625 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:39945
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:24:59.762494 335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:24:59.762872 335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:24:59.763084 335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:24:59.770771 335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:24:59.770892 335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:24:59.771136 335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:24:59.781405 335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.193.keytab
--principal=kudu/127.12.244.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:39945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:24:59.782819 335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:24:59.784247 335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:24:59.797820 340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:59.797793 343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:24:59.797999 341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:24:59.799050 335 server_base.cc:1034] running on GCE node
I20260926 02:24:59.799813 335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:24:59.800874 335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:24:59.802114 335 hybrid_clock.cc:648] HybridClock initialized: now 1790389499802063 us; error 44 us; skew 500 ppm
I20260926 02:24:59.802449 335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:24:59 dist-test-slave-qfmw krb5kdc[32697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389499, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:24:59.805853 335 init.cc:377] Logged in from keytab as kudu/127.12.244.193@KRBTEST.COM (short username kudu)
I20260926 02:24:59.807595 335 webserver.cc:460] Webserver started at http://127.12.244.193:36473/ using document root <none> and password file <none>
I20260926 02:24:59.808280 335 fs_manager.cc:359] Metadata directory not provided
I20260926 02:24:59.808384 335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:24:59.808668 335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:24:59.811894 335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "7c9de9c52b6a44ab977865c7d85a3e7b"
format_stamp: "Formatted at 2026-09-26 02:24:59 on dist-test-slave-qfmw"
server_key: "4da0effc580d95a7b7fc514633cf1e3d"
server_key_iv: "9ee049feaa0fbc98c36a7d2fa3f29967"
server_key_version: "encryptionkey@0"
I20260926 02:24:59.812739 335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "7c9de9c52b6a44ab977865c7d85a3e7b"
format_stamp: "Formatted at 2026-09-26 02:24:59 on dist-test-slave-qfmw"
server_key: "4da0effc580d95a7b7fc514633cf1e3d"
server_key_iv: "9ee049feaa0fbc98c36a7d2fa3f29967"
server_key_version: "encryptionkey@0"
I20260926 02:24:59.818543 335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:24:59.822691 350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:59.824327 335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:24:59.824537 335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "7c9de9c52b6a44ab977865c7d85a3e7b"
format_stamp: "Formatted at 2026-09-26 02:24:59 on dist-test-slave-qfmw"
server_key: "4da0effc580d95a7b7fc514633cf1e3d"
server_key_iv: "9ee049feaa0fbc98c36a7d2fa3f29967"
server_key_version: "encryptionkey@0"
I20260926 02:24:59.824760 335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:24:59.839186 335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:24:59.843945 335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:24:59.844281 335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:24:59.845353 335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:24:59.846823 335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:24:59.846900 335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:59.847028 335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:24:59.847116 335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:24:59.882534 335 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:46245
I20260926 02:24:59.882647 462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:46245 every 8 connection(s)
I20260926 02:24:59.884147 335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:24:59.891575 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 335
I20260926 02:24:59.891774 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:24:59.892274 13267 external_mini_cluster.cc:1428] Setting key 678ac5d67227bf8d9dd67b6c19e53417
Sep 26 02:24:59 dist-test-slave-qfmw krb5kdc[32697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389499, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.193@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:24:59.908798 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:59.886938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:41505 (local address 127.12.244.254:39945)
0926 02:24:59.887504 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 02:24:59.887508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:24:59.888554 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 02:24:59.890510 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:24:59.890516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:24:59.890625 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:24:59.890756 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:24:59.892747 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:59.893805 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:59.896616 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:24:59.897056 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:24:59.897170 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:59.902261 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:24:59.902271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:24:59.902279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:24:59.902334 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:24:59.905298 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:59.906174 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:59.906179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:59.906181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:59.906351 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:24:59.906832 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:24:59.906836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:24:59.906840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:24:59.907207 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:24:59.907367 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 02:24:59.908140 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 02:24:59.908425 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":303,"threads_started":1}
I20260926 02:24:59.909823 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:24:59.887489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39945 (local address 127.12.244.193:41505)
0926 02:24:59.888311 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 02:24:59.888353 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:24:59.890192 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:24:59.891032 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:24:59.891042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:24:59.891790 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:24:59.892542 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:59.892564 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:59.894013 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:59.894020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:59.896377 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:24:59.896402 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:24:59.897327 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:24:59.897337 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:24:59.897694 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:24:59.898589 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:24:59.898625 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:24:59.901902 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:24:59.905502 (+ 3600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:59.905513 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:59.905533 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:59.906009 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:59.906470 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:24:59.906475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:24:59.906479 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:24:59.906702 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:24:59.907340 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:24:59.907349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:24:59.907737 (+ 388us) client_negotiation.cc:769] Sending connection context
0926 02:24:59.908015 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 02:24:59.908628 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":284,"threads_started":1}
I20260926 02:24:59.911813 463 heartbeater.cc:347] Connected to a master server at 127.12.244.254:39945
I20260926 02:24:59.912194 463 heartbeater.cc:464] Registering TS with master...
I20260926 02:24:59.913061 463 heartbeater.cc:511] Master 127.12.244.254:39945 requested a full tablet report, sending...
I20260926 02:24:59.915709 32742 ts_manager.cc:194] Registered new tserver with Master: 7c9de9c52b6a44ab977865c7d85a3e7b (127.12.244.193:46245)
I20260926 02:24:59.917369 32742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.193@KRBTEST.COM'} at 127.12.244.193:41505
WARNING: no policy specified for kudu/127.12.244.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
Entry for principal kudu/127.12.244.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab.
I20260926 02:24:59.955341 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:39945
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:00.307443 469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:00.307822 469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:00.307996 469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:00.315611 469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:00.315786 469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:00.315980 469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:25:00.326758 469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.194.keytab
--principal=kudu/127.12.244.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:39945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:00.328212 469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:00.329660 469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:00.343305 477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:00.343430 474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:00.343430 475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:00.344519 469 server_base.cc:1034] running on GCE node
I20260926 02:25:00.345219 469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:00.346248 469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:00.347478 469 hybrid_clock.cc:648] HybridClock initialized: now 1790389500347431 us; error 47 us; skew 500 ppm
I20260926 02:25:00.347852 469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:00 dist-test-slave-qfmw krb5kdc[32697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389500, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:00.351440 469 init.cc:377] Logged in from keytab as kudu/127.12.244.194@KRBTEST.COM (short username kudu)
I20260926 02:25:00.353207 469 webserver.cc:460] Webserver started at http://127.12.244.194:42167/ using document root <none> and password file <none>
I20260926 02:25:00.353965 469 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:00.354105 469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:00.354440 469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:00.357604 469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "147cdce14fb947dfab1e7df88cbbce8b"
format_stamp: "Formatted at 2026-09-26 02:25:00 on dist-test-slave-qfmw"
server_key: "3869197537e05ff8c60a2c8677d6fbfb"
server_key_iv: "9f9876ec1b16e9dc6bd0e446a385c9ef"
server_key_version: "encryptionkey@0"
I20260926 02:25:00.358438 469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "147cdce14fb947dfab1e7df88cbbce8b"
format_stamp: "Formatted at 2026-09-26 02:25:00 on dist-test-slave-qfmw"
server_key: "3869197537e05ff8c60a2c8677d6fbfb"
server_key_iv: "9f9876ec1b16e9dc6bd0e446a385c9ef"
server_key_version: "encryptionkey@0"
I20260926 02:25:00.364375 469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:25:00.368129 484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:00.369619 469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:25:00.369817 469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "147cdce14fb947dfab1e7df88cbbce8b"
format_stamp: "Formatted at 2026-09-26 02:25:00 on dist-test-slave-qfmw"
server_key: "3869197537e05ff8c60a2c8677d6fbfb"
server_key_iv: "9f9876ec1b16e9dc6bd0e446a385c9ef"
server_key_version: "encryptionkey@0"
I20260926 02:25:00.370042 469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:00.382987 469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:00.387666 469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:00.387980 469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:00.389032 469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:25:00.390472 469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:25:00.390585 469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:00.390731 469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:25:00.390802 469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:00.424630 469 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:45157
I20260926 02:25:00.424721 597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:45157 every 8 connection(s)
I20260926 02:25:00.425997 469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:25:00.432647 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 469
I20260926 02:25:00.432845 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:25:00.433279 13267 external_mini_cluster.cc:1428] Setting key 1243335f1dca75d2ec2006ac5dfcd1d1
Sep 26 02:25:00 dist-test-slave-qfmw krb5kdc[32697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389500, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.194@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:00.449950 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:00.428692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:46471 (local address 127.12.244.254:39945)
0926 02:25:00.429144 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:00.429150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:00.430201 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 02:25:00.432158 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:00.432163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:00.432293 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:00.432434 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:00.434314 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:00.435410 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:00.438098 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:00.438511 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:00.438630 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:00.443037 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:00.443042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:00.443045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:00.443130 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:00.445917 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:00.446951 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:00.446955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:00.446958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:00.447196 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:00.447826 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:00.447830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:00.447833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:00.448164 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:00.448295 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:00.449371 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0926 02:25:00.449637 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":201,"threads_started":1}
I20260926 02:25:00.451474 601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:00.429291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39945 (local address 127.12.244.194:46471)
0926 02:25:00.430013 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:00.430071 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:00.431857 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:00.432695 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:00.432704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:00.433393 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:00.434081 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:00.434117 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:00.435578 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:00.435584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:00.437917 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:00.437936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:00.438753 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:00.438762 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:00.439143 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:00.439941 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:00.439979 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:00.442761 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:00.446084 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:00.446097 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:00.446122 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:00.446805 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:00.447327 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:00.447335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:00.447342 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:00.447679 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:00.448326 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:00.448336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:00.448888 (+ 552us) client_negotiation.cc:769] Sending connection context
0926 02:25:00.449247 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 02:25:00.449893 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":291,"threads_started":1}
I20260926 02:25:00.453338 598 heartbeater.cc:347] Connected to a master server at 127.12.244.254:39945
I20260926 02:25:00.453821 598 heartbeater.cc:464] Registering TS with master...
I20260926 02:25:00.454645 598 heartbeater.cc:511] Master 127.12.244.254:39945 requested a full tablet report, sending...
I20260926 02:25:00.456411 32742 ts_manager.cc:194] Registered new tserver with Master: 147cdce14fb947dfab1e7df88cbbce8b (127.12.244.194:45157)
I20260926 02:25:00.457731 32742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.194@KRBTEST.COM'} at 127.12.244.194:46471
WARNING: no policy specified for kudu/127.12.244.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
Entry for principal kudu/127.12.244.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab.
I20260926 02:25:00.502138 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:39945
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:00.851843 604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:00.852200 604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:00.852377 604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:00.860150 604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:00.860306 604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:00.860461 604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:25:00.871244 604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.195.keytab
--principal=kudu/127.12.244.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:39945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:00.872689 604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:00.874114 604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:00.888609 610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:00.888633 612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:00.888842 609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:00.889509 604 server_base.cc:1034] running on GCE node
I20260926 02:25:00.890170 604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:00.891291 604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:00.892565 604 hybrid_clock.cc:648] HybridClock initialized: now 1790389500892502 us; error 49 us; skew 500 ppm
I20260926 02:25:00.892938 604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:00 dist-test-slave-qfmw krb5kdc[32697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389500, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:00.903407 604 init.cc:377] Logged in from keytab as kudu/127.12.244.195@KRBTEST.COM (short username kudu)
I20260926 02:25:00.905167 604 webserver.cc:460] Webserver started at http://127.12.244.195:37443/ using document root <none> and password file <none>
I20260926 02:25:00.906034 604 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:00.906204 604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:00.906554 604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:00.909799 604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "835f2f6326df4f7fbf9995a5b7d35d31"
format_stamp: "Formatted at 2026-09-26 02:25:00 on dist-test-slave-qfmw"
server_key: "8aa958d4296bc135e781b9f052909fb6"
server_key_iv: "f4c30ee86866f6123bc11a1210112ada"
server_key_version: "encryptionkey@0"
I20260926 02:25:00.910691 604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "835f2f6326df4f7fbf9995a5b7d35d31"
format_stamp: "Formatted at 2026-09-26 02:25:00 on dist-test-slave-qfmw"
server_key: "8aa958d4296bc135e781b9f052909fb6"
server_key_iv: "f4c30ee86866f6123bc11a1210112ada"
server_key_version: "encryptionkey@0"
I20260926 02:25:00.917606 604 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:25:00.920955 463 heartbeater.cc:503] Master 127.12.244.254:39945 was elected leader, sending a full tablet report...
I20260926 02:25:00.921804 619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:00.923655 604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:25:00.923863 604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "835f2f6326df4f7fbf9995a5b7d35d31"
format_stamp: "Formatted at 2026-09-26 02:25:00 on dist-test-slave-qfmw"
server_key: "8aa958d4296bc135e781b9f052909fb6"
server_key_iv: "f4c30ee86866f6123bc11a1210112ada"
server_key_version: "encryptionkey@0"
I20260926 02:25:00.924108 604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:00.949370 604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:00.954876 604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:00.955245 604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:00.956367 604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:25:00.957841 604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:25:00.957918 604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:00.958093 604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:25:00.958159 604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:00.993075 604 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:44653
I20260926 02:25:00.993136 733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:44653 every 8 connection(s)
I20260926 02:25:00.994521 604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:25:01.002205 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 604
I20260926 02:25:01.002472 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:25:01.002979 13267 external_mini_cluster.cc:1428] Setting key a08372fe0341eb1fcdab93da78bab59c
Sep 26 02:25:01 dist-test-slave-qfmw krb5kdc[32697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389500, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.195@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:01.017903 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:00.997443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:58211 (local address 127.12.244.254:39945)
0926 02:25:00.997967 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:00.997972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:00.998890 (+ 918us) server_negotiation.cc:407] Connection header received
0926 02:25:01.000924 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.000930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.001323 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.001496 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:01.003644 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.004834 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.006718 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.007290 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.007417 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.011168 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:01.011178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:01.011185 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:01.011229 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:01.014125 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.015065 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.015071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.015073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.015284 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.015776 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.015779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.015781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.016077 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:01.016197 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.017163 (+ 966us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.017428 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
I20260926 02:25:01.019193 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:00.997921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39945 (local address 127.12.244.195:58211)
0926 02:25:00.998709 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:00.998750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:01.000598 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:01.001808 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:01.001818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:01.002518 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:01.003361 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.003384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.005034 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.005046 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.006554 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.006567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.007503 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.007508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.007737 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.008463 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:01.008488 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:01.010840 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:01.014314 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:01.014323 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:01.014340 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:01.014850 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:01.015411 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:01.015414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:01.015417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:01.015666 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:01.016214 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:01.016221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:01.016708 (+ 487us) client_negotiation.cc:769] Sending connection context
0926 02:25:01.017046 (+ 338us) client_negotiation.cc:240] Negotiation successful
0926 02:25:01.017604 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":293,"threads_started":1}
I20260926 02:25:01.020689 734 heartbeater.cc:347] Connected to a master server at 127.12.244.254:39945
I20260926 02:25:01.021049 734 heartbeater.cc:464] Registering TS with master...
I20260926 02:25:01.021797 734 heartbeater.cc:511] Master 127.12.244.254:39945 requested a full tablet report, sending...
I20260926 02:25:01.023420 32742 ts_manager.cc:194] Registered new tserver with Master: 835f2f6326df4f7fbf9995a5b7d35d31 (127.12.244.195:44653)
I20260926 02:25:01.024235 32742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.244.195@KRBTEST.COM'} at 127.12.244.195:58211
I20260926 02:25:01.032346 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:25:01.050995 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.036844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.12.244.254:39945)
0926 02:25:01.037060 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.037067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.037444 (+ 377us) server_negotiation.cc:407] Connection header received
0926 02:25:01.037675 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.037679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.037762 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.037878 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:01.039383 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.040141 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.041611 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.042045 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.042199 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.044934 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:01.044939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:01.044942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:01.044984 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:01.047899 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.048730 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.048736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.048741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.048892 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.049329 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.049334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.049339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.049710 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:01.049979 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.050401 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.050663 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:25:01.055667 32742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:25:01.063287 32742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:25:01.089288 747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.077451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45157 (local address 127.0.0.1:44222)
0926 02:25:01.078205 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:01.078247 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:01.078432 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:01.079221 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:01.079228 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:01.079277 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:01.080107 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.080124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.083314 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.083319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.085899 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.085915 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.087945 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.087952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.088206 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.088321 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 02:25:01.088497 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 02:25:01.088775 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":89,"thread_start_us":312,"threads_started":1}
I20260926 02:25:01.089783 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.077678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.12.244.194:45157)
0926 02:25:01.078299 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.078310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.078341 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:25:01.078646 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.078657 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.078946 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.080327 (+ 1381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:01.080588 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.083147 (+ 2559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.086127 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.087711 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.087889 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.088110 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.088564 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.089472 (+ 908us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":312,"threads_started":1}
I20260926 02:25:01.089990 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.078813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49020 (local address 127.12.244.193:46245)
0926 02:25:01.079760 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.079769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.079796 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:25:01.079890 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.079899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.080145 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.081045 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:01.081289 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.082884 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.086707 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.088031 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.088240 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.088409 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.089489 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.089673 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":581,"threads_started":1}
I20260926 02:25:01.089340 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.078461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:46245 (local address 127.0.0.1:49020)
0926 02:25:01.078990 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:01.079008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:01.079163 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:01.080348 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:01.080352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:01.080378 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:01.080830 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.080842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.083055 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.083060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.086488 (+ 3428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.086502 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.088277 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.088286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.088522 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.088568 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 02:25:01.088676 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 02:25:01.088843 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":921,"thread_start_us":245,"threads_started":1}
I20260926 02:25:01.093766 532 tablet_service.cc:1467] Processing CreateTablet for tablet 98021d5b75914f9b83a1932b3ad6a620 (DEFAULT_TABLE table=test-table [id=251fccc2e2d6424c8a203f121bc861eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:25:01.094132 398 tablet_service.cc:1467] Processing CreateTablet for tablet 98021d5b75914f9b83a1932b3ad6a620 (DEFAULT_TABLE table=test-table [id=251fccc2e2d6424c8a203f121bc861eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:25:01.095520 532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98021d5b75914f9b83a1932b3ad6a620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:01.095605 398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98021d5b75914f9b83a1932b3ad6a620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:01.096071 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.079767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44653 (local address 127.0.0.1:44742)
0926 02:25:01.084945 (+ 5178us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:01.084965 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:01.085091 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:01.088149 (+ 3058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:01.088155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:01.088190 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:01.088713 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.088725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.090721 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.090726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.092850 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.092869 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.095154 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.095163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.095349 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.095384 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:25:01.095481 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 02:25:01.095659 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5031,"mutex_wait_us":30,"thread_start_us":4855,"threads_started":1}
I20260926 02:25:01.096375 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.080103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.12.244.195:44653)
0926 02:25:01.087153 (+ 7050us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.087165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.087202 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:25:01.087303 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.087311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.087621 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.087851 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:01.088934 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.090535 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.093085 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.094567 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.094787 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.094970 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.095842 (+ 872us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.096024 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6817,"thread_start_us":6661,"threads_started":1}
I20260926 02:25:01.100674 668 tablet_service.cc:1467] Processing CreateTablet for tablet 98021d5b75914f9b83a1932b3ad6a620 (DEFAULT_TABLE table=test-table [id=251fccc2e2d6424c8a203f121bc861eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:25:01.102355 668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98021d5b75914f9b83a1932b3ad6a620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:01.108402 753 tablet_bootstrap.cc:492] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Bootstrap starting.
I20260926 02:25:01.110869 754 tablet_bootstrap.cc:492] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Bootstrap starting.
I20260926 02:25:01.112522 753 tablet_bootstrap.cc:654] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:01.114070 753 log.cc:824] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:01.115032 754 tablet_bootstrap.cc:654] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:01.116349 754 log.cc:824] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:01.117419 756 tablet_bootstrap.cc:492] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Bootstrap starting.
I20260926 02:25:01.118002 753 tablet_bootstrap.cc:492] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: No bootstrap required, opened a new log
I20260926 02:25:01.118472 753 ts_tablet_manager.cc:1397] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260926 02:25:01.120882 756 tablet_bootstrap.cc:654] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:01.121454 754 tablet_bootstrap.cc:492] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: No bootstrap required, opened a new log
I20260926 02:25:01.121763 754 ts_tablet_manager.cc:1397] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260926 02:25:01.122133 756 log.cc:824] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:01.126142 756 tablet_bootstrap.cc:492] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: No bootstrap required, opened a new log
I20260926 02:25:01.126461 756 ts_tablet_manager.cc:1397] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 02:25:01.129966 753 raft_consensus.cc:348] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.130570 753 raft_consensus.cc:374] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:01.130714 753 raft_consensus.cc:729] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 147cdce14fb947dfab1e7df88cbbce8b, State: Initialized, Role: FOLLOWER
I20260926 02:25:01.131508 753 consensus_queue.cc:260] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.132863 598 heartbeater.cc:503] Master 127.12.244.254:39945 was elected leader, sending a full tablet report...
I20260926 02:25:01.133844 753 ts_tablet_manager.cc:1428] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 02:25:01.135495 754 raft_consensus.cc:348] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.135663 756 raft_consensus.cc:348] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.135962 754 raft_consensus.cc:374] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:01.136109 754 raft_consensus.cc:729] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9de9c52b6a44ab977865c7d85a3e7b, State: Initialized, Role: FOLLOWER
I20260926 02:25:01.136139 756 raft_consensus.cc:374] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:01.136245 756 raft_consensus.cc:729] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835f2f6326df4f7fbf9995a5b7d35d31, State: Initialized, Role: FOLLOWER
I20260926 02:25:01.136932 754 consensus_queue.cc:260] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.136977 756 consensus_queue.cc:260] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.141734 754 ts_tablet_manager.cc:1428] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s
I20260926 02:25:01.143232 734 heartbeater.cc:503] Master 127.12.244.254:39945 was elected leader, sending a full tablet report...
I20260926 02:25:01.144097 756 ts_tablet_manager.cc:1428] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
W20260926 02:25:01.178155 599 tablet.cc:2369] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:25:01.246356 735 tablet.cc:2369] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:25:01.340849 760 raft_consensus.cc:482] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:25:01.341230 760 raft_consensus.cc:504] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.343361 760 leader_election.cc:288] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 147cdce14fb947dfab1e7df88cbbce8b (127.12.244.194:45157), 835f2f6326df4f7fbf9995a5b7d35d31 (127.12.244.195:44653)
I20260926 02:25:01.350056 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.343708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:40443 (local address 127.12.244.194:45157)
0926 02:25:01.344035 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.344043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.344511 (+ 468us) server_negotiation.cc:407] Connection header received
0926 02:25:01.344872 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.344877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.344937 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.345041 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:01.345587 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.346304 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.348231 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.349020 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.349234 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.349323 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.349608 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.349761 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260926 02:25:01.350227 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.343563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:45157 (local address 127.12.244.193:40443)
0926 02:25:01.344369 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:01.344395 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:01.344702 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:01.345070 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:01.345075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:01.345100 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:01.345440 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.345449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.346445 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.346451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.348077 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.348090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.349279 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.349285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.349443 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.349469 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 02:25:01.349585 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 02:25:01.349738 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":3,"thread_start_us":366,"threads_started":1}
I20260926 02:25:01.350656 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.344384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:56461 (local address 127.12.244.195:44653)
0926 02:25:01.344621 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.344629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.344781 (+ 152us) server_negotiation.cc:407] Connection header received
0926 02:25:01.345023 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.345028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.345116 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.345216 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:01.345913 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.346705 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.348677 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.349543 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.349773 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.349864 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.350212 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.350359 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 02:25:01.350807 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.343706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44653 (local address 127.12.244.193:56461)
0926 02:25:01.344658 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:01.344684 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:01.344860 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:01.345266 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:01.345277 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:01.345304 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:01.345736 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.345745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.346891 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.346899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.348499 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:01.348513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.349802 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.349809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:01.350013 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.350039 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 02:25:01.350180 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 02:25:01.350339 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":20,"thread_start_us":593,"threads_started":1}
I20260926 02:25:01.351809 552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98021d5b75914f9b83a1932b3ad6a620" candidate_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147cdce14fb947dfab1e7df88cbbce8b" is_pre_election: true
I20260926 02:25:01.352262 688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98021d5b75914f9b83a1932b3ad6a620" candidate_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" is_pre_election: true
I20260926 02:25:01.352545 552 raft_consensus.cc:2454] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c9de9c52b6a44ab977865c7d85a3e7b in term 0.
I20260926 02:25:01.352807 688 raft_consensus.cc:2454] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c9de9c52b6a44ab977865c7d85a3e7b in term 0.
I20260926 02:25:01.353459 354 leader_election.cc:302] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 147cdce14fb947dfab1e7df88cbbce8b, 7c9de9c52b6a44ab977865c7d85a3e7b; no voters:
I20260926 02:25:01.353919 760 raft_consensus.cc:2781] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:25:01.354097 760 raft_consensus.cc:482] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:25:01.354218 760 raft_consensus.cc:3037] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:01.356760 760 raft_consensus.cc:504] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.357522 760 leader_election.cc:288] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [CANDIDATE]: Term 1 election: Requested vote from peers 147cdce14fb947dfab1e7df88cbbce8b (127.12.244.194:45157), 835f2f6326df4f7fbf9995a5b7d35d31 (127.12.244.195:44653)
I20260926 02:25:01.358104 552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98021d5b75914f9b83a1932b3ad6a620" candidate_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "147cdce14fb947dfab1e7df88cbbce8b"
I20260926 02:25:01.358228 688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98021d5b75914f9b83a1932b3ad6a620" candidate_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835f2f6326df4f7fbf9995a5b7d35d31"
I20260926 02:25:01.358352 552 raft_consensus.cc:3037] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:01.358479 688 raft_consensus.cc:3037] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:01.361083 688 raft_consensus.cc:2454] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c9de9c52b6a44ab977865c7d85a3e7b in term 1.
I20260926 02:25:01.361213 552 raft_consensus.cc:2454] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c9de9c52b6a44ab977865c7d85a3e7b in term 1.
I20260926 02:25:01.361681 353 leader_election.cc:302] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c9de9c52b6a44ab977865c7d85a3e7b, 835f2f6326df4f7fbf9995a5b7d35d31; no voters:
I20260926 02:25:01.362083 760 raft_consensus.cc:2781] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:01.362589 760 raft_consensus.cc:686] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 1 LEADER]: Becoming Leader. State: Replica: 7c9de9c52b6a44ab977865c7d85a3e7b, State: Running, Role: LEADER
I20260926 02:25:01.363240 760 consensus_queue.cc:237] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } }
I20260926 02:25:01.368096 32742 catalog_manager.cc:5495] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c9de9c52b6a44ab977865c7d85a3e7b (127.12.244.193). New cstate: current_term: 1 leader_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c9de9c52b6a44ab977865c7d85a3e7b" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 } health_report { overall_health: UNKNOWN } } }
W20260926 02:25:01.387039 464 tablet.cc:2369] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:25:01.393515 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.387231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.12.244.193:46245)
0926 02:25:01.387474 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.387479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.387499 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:25:01.387721 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.387725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.387820 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.387945 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:01.388481 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.389230 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.390666 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.391083 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.391257 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.392019 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:01.392690 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:01.392815 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.393049 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.393134 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 02:25:01.403031 552 raft_consensus.cc:1261] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 1 FOLLOWER]: Refusing update from remote peer 7c9de9c52b6a44ab977865c7d85a3e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:25:01.403046 688 raft_consensus.cc:1261] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 1 FOLLOWER]: Refusing update from remote peer 7c9de9c52b6a44ab977865c7d85a3e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:25:01.404223 760 consensus_queue.cc:1035] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "147cdce14fb947dfab1e7df88cbbce8b" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 45157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:01.404593 764 consensus_queue.cc:1035] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "835f2f6326df4f7fbf9995a5b7d35d31" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:01.417651 767 mvcc.cc:205] Tried to move back new op lower bound from 7333435397734961152 to 7333435397588172800. Current Snapshot: MvccSnapshot[applied={T|T < 7333435397734961152}]
I20260926 02:25:01.442348 32741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:25:01.442672 32741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:25:01.447398 32741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fb9dcb7e91546738c2045e7592b0134: Sending DeleteTablet for 3 replicas of tablet 98021d5b75914f9b83a1932b3ad6a620
I20260926 02:25:01.448823 532 tablet_service.cc:1514] Processing DeleteTablet for tablet 98021d5b75914f9b83a1932b3ad6a620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:25:01 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:44222
I20260926 02:25:01.449616 398 tablet_service.cc:1514] Processing DeleteTablet for tablet 98021d5b75914f9b83a1932b3ad6a620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:25:01 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:49020
I20260926 02:25:01.449795 777 tablet_replica.cc:330] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: stopping tablet replica
I20260926 02:25:01.450222 777 raft_consensus.cc:2229] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:25:01.450770 777 raft_consensus.cc:2258] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:25:01.452410 778 tablet_replica.cc:330] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: stopping tablet replica
I20260926 02:25:01.452819 668 tablet_service.cc:1514] Processing DeleteTablet for tablet 98021d5b75914f9b83a1932b3ad6a620 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:25:01 UTC) from {username='kudu', principal='kudu/127.12.244.254@KRBTEST.COM'} at 127.0.0.1:44742
I20260926 02:25:01.452906 778 raft_consensus.cc:2229] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:25:01.453244 777 ts_tablet_manager.cc:1905] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:25:01.453578 778 raft_consensus.cc:2258] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:25:01.454643 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 335
I20260926 02:25:01.461463 778 ts_tablet_manager.cc:1905] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:25:01.462199 777 ts_tablet_manager.cc:1918] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:25:01.462486 777 log.cc:1196] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/98021d5b75914f9b83a1932b3ad6a620
I20260926 02:25:01.463071 777 ts_tablet_manager.cc:1939] T 98021d5b75914f9b83a1932b3ad6a620 P 147cdce14fb947dfab1e7df88cbbce8b: Deleting consensus metadata
I20260926 02:25:01.465728 32729 catalog_manager.cc:4841] TS 147cdce14fb947dfab1e7df88cbbce8b (127.12.244.194:45157): tablet 98021d5b75914f9b83a1932b3ad6a620 (table test-table [id=251fccc2e2d6424c8a203f121bc861eb]) successfully deleted
I20260926 02:25:01.465890 779 tablet_replica.cc:330] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: stopping tablet replica
I20260926 02:25:01.466367 779 raft_consensus.cc:2229] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:25:01.466948 779 raft_consensus.cc:2258] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:25:01.469360 779 ts_tablet_manager.cc:1905] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:25:01.473024 778 ts_tablet_manager.cc:1918] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:25:01.473292 778 log.cc:1196] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/98021d5b75914f9b83a1932b3ad6a620
I20260926 02:25:01.473997 778 ts_tablet_manager.cc:1939] T 98021d5b75914f9b83a1932b3ad6a620 P 7c9de9c52b6a44ab977865c7d85a3e7b: Deleting consensus metadata
I20260926 02:25:01.476094 32727 catalog_manager.cc:4841] TS 7c9de9c52b6a44ab977865c7d85a3e7b (127.12.244.193:46245): tablet 98021d5b75914f9b83a1932b3ad6a620 (table test-table [id=251fccc2e2d6424c8a203f121bc861eb]) successfully deleted
I20260926 02:25:01.476755 779 ts_tablet_manager.cc:1918] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:25:01.476974 779 log.cc:1196] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/98021d5b75914f9b83a1932b3ad6a620
I20260926 02:25:01.477449 779 ts_tablet_manager.cc:1939] T 98021d5b75914f9b83a1932b3ad6a620 P 835f2f6326df4f7fbf9995a5b7d35d31: Deleting consensus metadata
I20260926 02:25:01.479075 32728 catalog_manager.cc:4841] TS 835f2f6326df4f7fbf9995a5b7d35d31 (127.12.244.195:44653): tablet 98021d5b75914f9b83a1932b3ad6a620 (table test-table [id=251fccc2e2d6424c8a203f121bc861eb]) successfully deleted
Sep 26 02:25:01 dist-test-slave-qfmw krb5kdc[32697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.193@KRBTEST.COM
I20260926 02:25:01.488808 750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.456277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.12.244.193:46245)
0926 02:25:01.456476 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.456483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.467399 (+ 10916us) server_negotiation.cc:407] Connection header received
0926 02:25:01.467720 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.467726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.467819 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.467895 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:01.469791 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.471168 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.474559 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.475324 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.475445 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.481152 (+ 5707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:01.481173 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:01.481194 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:01.481245 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:01.485479 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.486322 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.486332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.486336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.486536 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.486935 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.486941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.486945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.487492 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:01.487814 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.488142 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.488406 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:25:01.489708 458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:01.591475 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 335
I20260926 02:25:01.611521 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 469
Sep 26 02:25:01 dist-test-slave-qfmw krb5kdc[32697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.194@KRBTEST.COM
I20260926 02:25:01.628245 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.612426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.12.244.194:45157)
0926 02:25:01.612632 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.612636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.612656 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:25:01.612712 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.612716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.612779 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.612864 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:01.614275 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.615340 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.616715 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.617142 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.617254 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.621920 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:01.621939 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:01.621956 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:01.622001 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:01.625105 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.625940 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.625948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.625951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.626094 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.626487 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.626490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.626492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.626899 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:01.627195 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.627696 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.627871 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 02:25:01.629065 593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:01.721886 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 469
I20260926 02:25:01.739603 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 604
Sep 26 02:25:01 dist-test-slave-qfmw krb5kdc[32697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.195@KRBTEST.COM
I20260926 02:25:01.755828 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:01.740486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36752 (local address 127.12.244.195:44653)
0926 02:25:01.740677 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:01.740682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:01.740752 (+ 70us) server_negotiation.cc:407] Connection header received
0926 02:25:01.740931 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:01.740935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:01.741016 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:01.741101 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:01.742504 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.743307 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.744576 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:01.744973 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:01.745074 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:01.749687 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:01.749705 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:01.749723 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:01.749761 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:01.752789 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.753622 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.753630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.753632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.753779 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:01.754119 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:01.754122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:01.754124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:01.754500 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:01.754722 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:01.755150 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 02:25:01.755386 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:25:01.756615 729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:01.847827 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 604
I20260926 02:25:01.864869 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 32710
I20260926 02:25:01.866037 304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:01.981149 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 32710
2026-09-26T02:25:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4627 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T02:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:02Z Disabled control of system clock
I20260926 02:25:02.062495 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45669
--webserver_interface=127.12.244.254
--webserver_port=0
--builtin_ntp_servers=127.12.244.212:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:45669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:25:02.421656 788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:02.421958 788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:02.422048 788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:02.429611 788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:02.429746 788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:02.429793 788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:02.429850 788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:02.429901 788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:02.440711 788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:45669
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:02.442179 788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:02.444326 788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:02.457839 793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:02.457803 794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:02.457751 796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:02.459199 788 server_base.cc:1034] running on GCE node
I20260926 02:25:02.460130 788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:02.461680 788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:02.462903 788 hybrid_clock.cc:648] HybridClock initialized: now 1790389502462811 us; error 93 us; skew 500 ppm
I20260926 02:25:02.463340 788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:25:02.466516 788 webserver.cc:460] Webserver started at http://127.12.244.254:36945/ using document root <none> and password file <none>
I20260926 02:25:02.467511 788 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:02.467633 788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:02.468077 788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:02.471662 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a"
format_stamp: "Formatted at 2026-09-26 02:25:02 on dist-test-slave-qfmw"
server_key: "7c7e88c0ed3058f1010478f6c28a2686"
server_key_iv: "25460a015ce5725bdbd93a38c6670db1"
server_key_version: "encryptionkey@0"
I20260926 02:25:02.472682 788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a"
format_stamp: "Formatted at 2026-09-26 02:25:02 on dist-test-slave-qfmw"
server_key: "7c7e88c0ed3058f1010478f6c28a2686"
server_key_iv: "25460a015ce5725bdbd93a38c6670db1"
server_key_version: "encryptionkey@0"
I20260926 02:25:02.478781 788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:25:02.482770 802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:02.484478 788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:25:02.484710 788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a"
format_stamp: "Formatted at 2026-09-26 02:25:02 on dist-test-slave-qfmw"
server_key: "7c7e88c0ed3058f1010478f6c28a2686"
server_key_iv: "25460a015ce5725bdbd93a38c6670db1"
server_key_version: "encryptionkey@0"
I20260926 02:25:02.484961 788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:02.510387 788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:02.511420 788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:02.511734 788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:02.531828 788 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:45669
I20260926 02:25:02.531852 853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:45669 every 8 connection(s)
I20260926 02:25:02.533250 788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:02.537190 854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:02.539779 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 788
I20260926 02:25:02.540020 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:02.540524 13267 external_mini_cluster.cc:1428] Setting key 5654a2eac71a72db2b2e52dce8a00cac
I20260926 02:25:02.551256 854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a: Bootstrap starting.
I20260926 02:25:02.555523 854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:02.557106 854 log.cc:824] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:02.557255 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:02.542676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.12.244.254:45669)
0926 02:25:02.543889 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:02.543906 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:02.543959 (+ 53us) server_negotiation.cc:407] Connection header received
0926 02:25:02.546457 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:02.546494 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:02.547203 (+ 709us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:02.547713 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:02.548297 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:02.550292 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:02.551803 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:02.552862 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:02.553103 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:02.554206 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:02.554216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:02.554236 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:02.554995 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:02.555149 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:02.555178 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:02.555317 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:02.555503 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 02:25:02.555878 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":434,"threads_started":1}
I20260926 02:25:02.561066 854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a: No bootstrap required, opened a new log
I20260926 02:25:02.569165 854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45669 } }
I20260926 02:25:02.569569 854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:02.569676 854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9c2464d0ee48f1b03b22a4b4e3ba0a, State: Initialized, Role: FOLLOWER
I20260926 02:25:02.570292 854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45669 } }
I20260926 02:25:02.570526 854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:02.570632 854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:02.570808 854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:02.573745 854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45669 } }
I20260926 02:25:02.574291 854 leader_election.cc:302] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d9c2464d0ee48f1b03b22a4b4e3ba0a; no voters:
I20260926 02:25:02.574859 854 leader_election.cc:288] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:02.575032 859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:02.575877 859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [term 1 LEADER]: Becoming Leader. State: Replica: 7d9c2464d0ee48f1b03b22a4b4e3ba0a, State: Running, Role: LEADER
I20260926 02:25:02.576483 859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45669 } }
I20260926 02:25:02.577234 854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:02.580138 861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d9c2464d0ee48f1b03b22a4b4e3ba0a. Latest consensus state: current_term: 1 leader_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45669 } } }
I20260926 02:25:02.580651 861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:02.581312 860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d9c2464d0ee48f1b03b22a4b4e3ba0a" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45669 } } }
I20260926 02:25:02.581837 860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:02.586575 867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:02.592469 867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:02.602137 867 catalog_manager.cc:1347] Generated new cluster ID: 8c5239732b864ea0a192ac430ff428e1
I20260926 02:25:02.602317 867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:02.616122 867 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:02.618552 867 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:02.627064 867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a: Generated new TSK 0
I20260926 02:25:02.628129 867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:02.647557 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.193:0
--local_ip_for_outbound_sockets=127.12.244.193
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45669
--builtin_ntp_servers=127.12.244.212:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 02:25:03.020826 879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:03.021178 879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:03.021343 879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:03.029029 879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:03.029165 879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:03.029367 879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.193
I20260926 02:25:03.039146 879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.244.193
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.193
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:03.040661 879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:03.042551 879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:03.058432 884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:03.058779 885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:03.058847 887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:03.060894 879 server_base.cc:1034] running on GCE node
I20260926 02:25:03.061651 879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:03.062906 879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:03.064193 879 hybrid_clock.cc:648] HybridClock initialized: now 1790389503064113 us; error 77 us; skew 500 ppm
I20260926 02:25:03.064633 879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:25:03.067549 879 webserver.cc:460] Webserver started at http://127.12.244.193:46823/ using document root <none> and password file <none>
I20260926 02:25:03.068359 879 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:03.068495 879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:03.068869 879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:03.072737 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data/instance:
uuid: "911dd2be54dd45d594ef00d563bdc3a5"
format_stamp: "Formatted at 2026-09-26 02:25:03 on dist-test-slave-qfmw"
server_key: "ab7f8e7e15ae68cd0c2f4e4fd5d189c9"
server_key_iv: "9b16bb1f40e9e974ec1d331f7a023ac8"
server_key_version: "encryptionkey@0"
I20260926 02:25:03.073725 879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance:
uuid: "911dd2be54dd45d594ef00d563bdc3a5"
format_stamp: "Formatted at 2026-09-26 02:25:03 on dist-test-slave-qfmw"
server_key: "ab7f8e7e15ae68cd0c2f4e4fd5d189c9"
server_key_iv: "9b16bb1f40e9e974ec1d331f7a023ac8"
server_key_version: "encryptionkey@0"
I20260926 02:25:03.080117 879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 02:25:03.084932 893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:03.086683 879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:25:03.086937 879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal
uuid: "911dd2be54dd45d594ef00d563bdc3a5"
format_stamp: "Formatted at 2026-09-26 02:25:03 on dist-test-slave-qfmw"
server_key: "ab7f8e7e15ae68cd0c2f4e4fd5d189c9"
server_key_iv: "9b16bb1f40e9e974ec1d331f7a023ac8"
server_key_version: "encryptionkey@0"
I20260926 02:25:03.087280 879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:03.114283 879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:03.115365 879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:03.115722 879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:03.116921 879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:25:03.118639 879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:25:03.118741 879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:03.118909 879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:25:03.118994 879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:03.155604 879 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.193:40387
I20260926 02:25:03.155649 1005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.193:40387 every 8 connection(s)
I20260926 02:25:03.157027 879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/data/info.pb
I20260926 02:25:03.165175 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 879
I20260926 02:25:03.165397 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal/instance
I20260926 02:25:03.165889 13267 external_mini_cluster.cc:1428] Setting key 8155a4543f8442e726056465fffba3e3
I20260926 02:25:03.168711 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.194:0
--local_ip_for_outbound_sockets=127.12.244.194
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45669
--builtin_ntp_servers=127.12.244.212:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:25:03.171064 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:03.159916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.193:48723 (local address 127.12.244.254:45669)
0926 02:25:03.160453 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:03.160458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:03.161729 (+ 1271us) server_negotiation.cc:407] Connection header received
0926 02:25:03.163815 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:03.163821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:03.163935 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:03.164078 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:03.165324 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.166192 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.167981 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.168413 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.168555 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:03.170174 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:03.170180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:03.170183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:03.170204 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:03.170244 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:03.170257 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:03.170361 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:03.170607 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 02:25:03.170723 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":289,"threads_started":1}
I20260926 02:25:03.172673 1009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:03.160476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45669 (local address 127.12.244.193:48723)
0926 02:25:03.161492 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:03.161538 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:03.163476 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:03.164356 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:03.164367 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:03.164471 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:03.165127 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:03.165146 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.166372 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.166376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:03.167822 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:03.167836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.168692 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.168703 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:03.169012 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:03.169804 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:25:03.169807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:03.169853 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:25:03.169944 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:03.170359 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:03.170375 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:03.170386 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:25:03.170644 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 02:25:03.171178 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":294,"threads_started":1}
I20260926 02:25:03.175176 1006 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45669
I20260926 02:25:03.175531 1006 heartbeater.cc:464] Registering TS with master...
I20260926 02:25:03.176396 1006 heartbeater.cc:511] Master 127.12.244.254:45669 requested a full tablet report, sending...
I20260926 02:25:03.179423 819 ts_manager.cc:194] Registered new tserver with Master: 911dd2be54dd45d594ef00d563bdc3a5 (127.12.244.193:40387)
I20260926 02:25:03.181028 819 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.193:48723
W20260926 02:25:03.543345 1010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:03.543797 1010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:03.543984 1010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:03.556571 1010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:03.556690 1010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:03.556845 1010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.194
I20260926 02:25:03.566569 1010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.244.194
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.194
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:03.568176 1010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:03.570027 1010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:03.584498 1016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:03.584400 1017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:03.584697 1019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:03.585274 1010 server_base.cc:1034] running on GCE node
I20260926 02:25:03.585951 1010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:03.587028 1010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:03.588289 1010 hybrid_clock.cc:648] HybridClock initialized: now 1790389503588251 us; error 38 us; skew 500 ppm
I20260926 02:25:03.588647 1010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:25:03.591409 1010 webserver.cc:460] Webserver started at http://127.12.244.194:41599/ using document root <none> and password file <none>
I20260926 02:25:03.592355 1010 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:03.592458 1010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:03.592825 1010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:03.596079 1010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data/instance:
uuid: "ee6d5cadda99473ca0acba5a0bece871"
format_stamp: "Formatted at 2026-09-26 02:25:03 on dist-test-slave-qfmw"
server_key: "ac437dce10995d9faf1f7a5985429d9c"
server_key_iv: "13c34db20af4b03c3d37c08bb1e1a570"
server_key_version: "encryptionkey@0"
I20260926 02:25:03.597011 1010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance:
uuid: "ee6d5cadda99473ca0acba5a0bece871"
format_stamp: "Formatted at 2026-09-26 02:25:03 on dist-test-slave-qfmw"
server_key: "ac437dce10995d9faf1f7a5985429d9c"
server_key_iv: "13c34db20af4b03c3d37c08bb1e1a570"
server_key_version: "encryptionkey@0"
I20260926 02:25:03.602849 1010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:25:03.606806 1025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:03.608623 1010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:25:03.608829 1010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal
uuid: "ee6d5cadda99473ca0acba5a0bece871"
format_stamp: "Formatted at 2026-09-26 02:25:03 on dist-test-slave-qfmw"
server_key: "ac437dce10995d9faf1f7a5985429d9c"
server_key_iv: "13c34db20af4b03c3d37c08bb1e1a570"
server_key_version: "encryptionkey@0"
I20260926 02:25:03.609112 1010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:03.626303 1010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:03.627429 1010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:03.627792 1010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:03.628949 1010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:25:03.630575 1010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:25:03.630682 1010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:03.630817 1010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:25:03.630913 1010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:03.665499 1010 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.194:44035
I20260926 02:25:03.665551 1137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.194:44035 every 8 connection(s)
I20260926 02:25:03.666970 1010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/data/info.pb
I20260926 02:25:03.669483 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1010
I20260926 02:25:03.669688 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal/instance
I20260926 02:25:03.670269 13267 external_mini_cluster.cc:1428] Setting key 866957e43ab377b585355073af68b7b6
I20260926 02:25:03.673171 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.244.195:0
--local_ip_for_outbound_sockets=127.12.244.195
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45669
--builtin_ntp_servers=127.12.244.212:38679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 02:25:03.681970 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:03.669864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.194:33111 (local address 127.12.244.254:45669)
0926 02:25:03.670093 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:03.670100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:03.671780 (+ 1680us) server_negotiation.cc:407] Connection header received
0926 02:25:03.673786 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:03.673795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:03.673896 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:03.674022 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:03.675445 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.676601 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.678532 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.679319 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.679475 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:03.680986 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:03.680995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:03.681003 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:03.681033 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:03.681080 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:03.681100 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:03.681208 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:03.681442 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 02:25:03.681578 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:25:03.682976 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:03.670409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45669 (local address 127.12.244.194:33111)
0926 02:25:03.671596 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:03.671643 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:03.673496 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:03.674315 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:03.674326 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:03.674493 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:03.675246 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:03.675268 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.676807 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.676812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:03.678371 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:03.678384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:03.679599 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:03.679605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:03.679828 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:03.680608 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:25:03.680611 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:03.680665 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:25:03.680754 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:03.681219 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:03.681229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:03.681235 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 02:25:03.681391 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 02:25:03.681788 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":386,"threads_started":1}
I20260926 02:25:03.684746 1138 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45669
I20260926 02:25:03.685107 1138 heartbeater.cc:464] Registering TS with master...
I20260926 02:25:03.685974 1138 heartbeater.cc:511] Master 127.12.244.254:45669 requested a full tablet report, sending...
I20260926 02:25:03.687536 819 ts_manager.cc:194] Registered new tserver with Master: ee6d5cadda99473ca0acba5a0bece871 (127.12.244.194:44035)
I20260926 02:25:03.688460 819 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.194:33111
W20260926 02:25:04.027812 1141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:04.028126 1141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:04.028280 1141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:04.035955 1141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:04.036078 1141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:04.036232 1141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.244.195
I20260926 02:25:04.046028 1141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.244.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.244.195
--webserver_port=0
--tserver_master_addrs=127.12.244.254:45669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.244.195
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:04.047453 1141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:04.049319 1141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:04.064436 1146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:04.064554 1147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:04.064469 1149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:04.069273 1141 server_base.cc:1034] running on GCE node
I20260926 02:25:04.070127 1141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:04.071384 1141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:04.072701 1141 hybrid_clock.cc:648] HybridClock initialized: now 1790389504072645 us; error 69 us; skew 500 ppm
I20260926 02:25:04.073143 1141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:25:04.076316 1141 webserver.cc:460] Webserver started at http://127.12.244.195:44567/ using document root <none> and password file <none>
I20260926 02:25:04.077184 1141 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:04.077334 1141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:04.077723 1141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:04.081215 1141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data/instance:
uuid: "7864fbd3fc9c46a491f8ad848066da22"
format_stamp: "Formatted at 2026-09-26 02:25:04 on dist-test-slave-qfmw"
server_key: "a9cfe6b5a9bf337a0ba5f66afa56735b"
server_key_iv: "734af1b90468069dfd0a162e878451ac"
server_key_version: "encryptionkey@0"
I20260926 02:25:04.082132 1141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance:
uuid: "7864fbd3fc9c46a491f8ad848066da22"
format_stamp: "Formatted at 2026-09-26 02:25:04 on dist-test-slave-qfmw"
server_key: "a9cfe6b5a9bf337a0ba5f66afa56735b"
server_key_iv: "734af1b90468069dfd0a162e878451ac"
server_key_version: "encryptionkey@0"
I20260926 02:25:04.088362 1141 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:25:04.092689 1155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:04.094198 1141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:04.094409 1141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal
uuid: "7864fbd3fc9c46a491f8ad848066da22"
format_stamp: "Formatted at 2026-09-26 02:25:04 on dist-test-slave-qfmw"
server_key: "a9cfe6b5a9bf337a0ba5f66afa56735b"
server_key_iv: "734af1b90468069dfd0a162e878451ac"
server_key_version: "encryptionkey@0"
I20260926 02:25:04.094738 1141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:04.125483 1141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:04.126381 1141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:04.126703 1141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:04.127792 1141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:25:04.129401 1141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:25:04.129527 1141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:04.129676 1141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:25:04.129767 1141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:04.166528 1141 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.195:44865
I20260926 02:25:04.166591 1268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.195:44865 every 8 connection(s)
I20260926 02:25:04.168025 1141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/data/info.pb
I20260926 02:25:04.173375 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1141
I20260926 02:25:04.173573 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal/instance
I20260926 02:25:04.174154 13267 external_mini_cluster.cc:1428] Setting key 83e5cc9f83951950218fdc40d07c5971
I20260926 02:25:04.184032 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.170962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:42973 (local address 127.12.244.254:45669)
0926 02:25:04.171257 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.171262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.172772 (+ 1510us) server_negotiation.cc:407] Connection header received
0926 02:25:04.175511 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.175520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.175628 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.175734 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:04.177280 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.178295 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.180760 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.181197 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.181315 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.182929 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:04.182935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:04.182938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:04.182960 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:04.182995 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:04.183009 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:04.183183 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.183519 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.183629 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 02:25:04.185220 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.171609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:45669 (local address 127.12.244.195:42973)
0926 02:25:04.172592 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:04.172631 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:04.175052 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:04.175978 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:04.175988 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:04.176143 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:04.177059 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.177087 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.178499 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.178503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.180563 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.180580 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.181441 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.181447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.181713 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.182555 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:25:04.182558 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:04.182608 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:25:04.182706 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:04.183192 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:04.183206 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:04.183218 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:25:04.183406 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 02:25:04.183924 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":305,"threads_started":1}
I20260926 02:25:04.185375 1006 heartbeater.cc:503] Master 127.12.244.254:45669 was elected leader, sending a full tablet report...
I20260926 02:25:04.187129 1269 heartbeater.cc:347] Connected to a master server at 127.12.244.254:45669
I20260926 02:25:04.187536 1269 heartbeater.cc:464] Registering TS with master...
I20260926 02:25:04.188488 1269 heartbeater.cc:511] Master 127.12.244.254:45669 requested a full tablet report, sending...
I20260926 02:25:04.190176 819 ts_manager.cc:194] Registered new tserver with Master: 7864fbd3fc9c46a491f8ad848066da22 (127.12.244.195:44865)
I20260926 02:25:04.191045 819 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.244.195:42973
I20260926 02:25:04.203576 13267 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:25:04.215193 1008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.208329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.12.244.254:45669)
0926 02:25:04.208563 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.208569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.209006 (+ 437us) server_negotiation.cc:407] Connection header received
0926 02:25:04.209163 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.209168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.209248 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.209355 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:04.209928 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.210696 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.212343 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.212866 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.213012 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.214320 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:04.214325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:04.214328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:04.214351 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:04.214389 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:04.214403 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:04.214486 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.214652 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.214741 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:25:04.219640 819 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:25:04.227540 819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:25:04.265834 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.241894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.12.244.193:40387)
0926 02:25:04.253603 (+ 11709us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.253616 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.255766 (+ 2150us) server_negotiation.cc:407] Connection header received
0926 02:25:04.256140 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.256148 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.256419 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.256644 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:04.257315 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.259043 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.262200 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.263721 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.263862 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.264050 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.265247 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.265445 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11481,"thread_start_us":11301,"threads_started":1}
I20260926 02:25:04.265941 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.242447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.12.244.194:44035)
0926 02:25:04.252769 (+ 10322us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.252784 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.255791 (+ 3007us) server_negotiation.cc:407] Connection header received
0926 02:25:04.256602 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.256610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.256861 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.257088 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:04.258968 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.260569 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.263357 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.264793 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.264977 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.265220 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.265438 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.265601 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10053,"thread_start_us":10341,"threads_started":1}
I20260926 02:25:04.265941 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.242113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:44035 (local address 127.0.0.1:34812)
0926 02:25:04.255571 (+ 13458us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:04.255631 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:04.256396 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:04.257332 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:04.257336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:04.257364 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:04.258763 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.258776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.260760 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.260765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.263140 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.263156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.265024 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.265032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.265219 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.265260 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 02:25:04.265383 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 02:25:04.265532 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12735,"mutex_wait_us":21,"thread_start_us":11852,"threads_started":1}
I20260926 02:25:04.266399 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.242804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.12.244.195:44865)
0926 02:25:04.254345 (+ 11541us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.254356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.256290 (+ 1934us) server_negotiation.cc:407] Connection header received
0926 02:25:04.256537 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.256548 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.256816 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.257050 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:04.257786 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.259485 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.261988 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.264102 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.264323 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.264565 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.265659 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.265844 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11354,"thread_start_us":11201,"threads_started":1}
I20260926 02:25:04.266832 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.242264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.195:44865 (local address 127.0.0.1:43528)
0926 02:25:04.256148 (+ 13884us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:04.256167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:04.256326 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:04.257038 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:04.257044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:04.257080 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:04.257546 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.257560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.259632 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.259643 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.261648 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.261675 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.264428 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.264435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.264825 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.264956 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 02:25:04.265149 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 02:25:04.265394 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13773,"mutex_wait_us":171,"thread_start_us":13591,"threads_started":1}
I20260926 02:25:04.266889 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.241795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:40387 (local address 127.0.0.1:46222)
0926 02:25:04.255568 (+ 13773us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:04.255626 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:04.255865 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:04.256651 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:04.256658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:04.256686 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:04.257062 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.257077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.259294 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.259299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.261954 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.261970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.264357 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.264366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.264830 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.264956 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 02:25:04.266299 (+ 1343us) client_negotiation.cc:240] Negotiation successful
0926 02:25:04.266474 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12707,"mutex_wait_us":120,"thread_start_us":11959,"threads_started":1}
I20260926 02:25:04.270763 941 tablet_service.cc:1467] Processing CreateTablet for tablet f2d4e3ced5f645c6a295f9277774d99b (DEFAULT_TABLE table=test-table [id=54b6c86638974c30874814aae83f0c3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:25:04.270761 1073 tablet_service.cc:1467] Processing CreateTablet for tablet f2d4e3ced5f645c6a295f9277774d99b (DEFAULT_TABLE table=test-table [id=54b6c86638974c30874814aae83f0c3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:25:04.272393 941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d4e3ced5f645c6a295f9277774d99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:04.272840 1073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d4e3ced5f645c6a295f9277774d99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:04.274250 1203 tablet_service.cc:1467] Processing CreateTablet for tablet f2d4e3ced5f645c6a295f9277774d99b (DEFAULT_TABLE table=test-table [id=54b6c86638974c30874814aae83f0c3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:25:04.275880 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d4e3ced5f645c6a295f9277774d99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:04.284930 1287 tablet_bootstrap.cc:492] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Bootstrap starting.
I20260926 02:25:04.289268 1288 tablet_bootstrap.cc:492] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Bootstrap starting.
I20260926 02:25:04.289320 1287 tablet_bootstrap.cc:654] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:04.290696 1287 log.cc:824] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:04.291026 1289 tablet_bootstrap.cc:492] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Bootstrap starting.
I20260926 02:25:04.293993 1289 tablet_bootstrap.cc:654] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:04.294025 1288 tablet_bootstrap.cc:654] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:04.295724 1288 log.cc:824] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:04.295801 1289 log.cc:824] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:04.297411 1287 tablet_bootstrap.cc:492] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: No bootstrap required, opened a new log
I20260926 02:25:04.297734 1287 ts_tablet_manager.cc:1397] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 02:25:04.298962 1289 tablet_bootstrap.cc:492] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: No bootstrap required, opened a new log
I20260926 02:25:04.299335 1289 ts_tablet_manager.cc:1397] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 02:25:04.299657 1288 tablet_bootstrap.cc:492] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: No bootstrap required, opened a new log
I20260926 02:25:04.299974 1288 ts_tablet_manager.cc:1397] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260926 02:25:04.305925 1287 raft_consensus.cc:348] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.306679 1287 raft_consensus.cc:374] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:04.306860 1287 raft_consensus.cc:729] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911dd2be54dd45d594ef00d563bdc3a5, State: Initialized, Role: FOLLOWER
I20260926 02:25:04.307850 1287 consensus_queue.cc:260] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.310679 1289 raft_consensus.cc:348] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.310693 1288 raft_consensus.cc:348] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.311214 1288 raft_consensus.cc:374] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:04.311269 1289 raft_consensus.cc:374] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:04.311376 1287 ts_tablet_manager.cc:1428] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 02:25:04.311378 1288 raft_consensus.cc:729] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6d5cadda99473ca0acba5a0bece871, State: Initialized, Role: FOLLOWER
I20260926 02:25:04.311425 1289 raft_consensus.cc:729] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7864fbd3fc9c46a491f8ad848066da22, State: Initialized, Role: FOLLOWER
I20260926 02:25:04.312225 1289 consensus_queue.cc:260] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.312232 1288 consensus_queue.cc:260] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.314242 1289 ts_tablet_manager.cc:1428] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 02:25:04.314416 1269 heartbeater.cc:503] Master 127.12.244.254:45669 was elected leader, sending a full tablet report...
I20260926 02:25:04.314544 1288 ts_tablet_manager.cc:1428] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 02:25:04.315073 1138 heartbeater.cc:503] Master 127.12.244.254:45669 was elected leader, sending a full tablet report...
I20260926 02:25:04.316368 1294 raft_consensus.cc:482] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:25:04.316677 1294 raft_consensus.cc:504] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.320029 1294 leader_election.cc:288] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 911dd2be54dd45d594ef00d563bdc3a5 (127.12.244.193:40387), ee6d5cadda99473ca0acba5a0bece871 (127.12.244.194:44035)
I20260926 02:25:04.325562 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.319725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:37589 (local address 127.12.244.193:40387)
0926 02:25:04.319893 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.319899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.319917 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:25:04.319989 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.319993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.320121 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.320227 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:04.320930 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.322042 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.323737 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.324612 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.324785 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.324859 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.325171 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.325299 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 02:25:04.325572 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.319165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.193:40387 (local address 127.12.244.195:37589)
0926 02:25:04.319339 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:04.319360 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:04.319480 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:04.320410 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:04.320414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:04.320435 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:04.320708 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.320722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.322171 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.322180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.323579 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.323591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.324768 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.324773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.324963 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.324987 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:25:04.325133 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 02:25:04.325274 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 02:25:04.326269 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.320821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.244.195:53427 (local address 127.12.244.194:44035)
0926 02:25:04.320999 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.321006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.321193 (+ 187us) server_negotiation.cc:407] Connection header received
0926 02:25:04.321342 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.321345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.321411 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.321481 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:25:04.322026 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.322719 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.324550 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.325353 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.325652 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.325762 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.325901 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.326022 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:25:04.326305 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.320386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.194:44035 (local address 127.12.244.195:53427)
0926 02:25:04.321096 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:04.321114 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:04.321235 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:04.321525 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:04.321530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:04.321554 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:25:04.321884 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.321894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.322866 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.322871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.324366 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:04.324379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.325526 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.325534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:04.325727 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.325759 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:25:04.325877 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 02:25:04.326012 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":397,"threads_started":1}
I20260926 02:25:04.326982 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d4e3ced5f645c6a295f9277774d99b" candidate_uuid: "7864fbd3fc9c46a491f8ad848066da22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911dd2be54dd45d594ef00d563bdc3a5" is_pre_election: true
I20260926 02:25:04.327186 1093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d4e3ced5f645c6a295f9277774d99b" candidate_uuid: "7864fbd3fc9c46a491f8ad848066da22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6d5cadda99473ca0acba5a0bece871" is_pre_election: true
I20260926 02:25:04.327647 961 raft_consensus.cc:2454] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7864fbd3fc9c46a491f8ad848066da22 in term 0.
I20260926 02:25:04.327646 1093 raft_consensus.cc:2454] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7864fbd3fc9c46a491f8ad848066da22 in term 0.
I20260926 02:25:04.328408 1156 leader_election.cc:302] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7864fbd3fc9c46a491f8ad848066da22, ee6d5cadda99473ca0acba5a0bece871; no voters:
I20260926 02:25:04.328848 1294 raft_consensus.cc:2781] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:25:04.329003 1294 raft_consensus.cc:482] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:25:04.329128 1294 raft_consensus.cc:3037] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:04.331574 1294 raft_consensus.cc:504] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.332320 1294 leader_election.cc:288] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [CANDIDATE]: Term 1 election: Requested vote from peers 911dd2be54dd45d594ef00d563bdc3a5 (127.12.244.193:40387), ee6d5cadda99473ca0acba5a0bece871 (127.12.244.194:44035)
I20260926 02:25:04.332917 961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d4e3ced5f645c6a295f9277774d99b" candidate_uuid: "7864fbd3fc9c46a491f8ad848066da22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911dd2be54dd45d594ef00d563bdc3a5"
I20260926 02:25:04.332988 1093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d4e3ced5f645c6a295f9277774d99b" candidate_uuid: "7864fbd3fc9c46a491f8ad848066da22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee6d5cadda99473ca0acba5a0bece871"
I20260926 02:25:04.333118 961 raft_consensus.cc:3037] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:04.333294 1093 raft_consensus.cc:3037] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:04.335739 961 raft_consensus.cc:2454] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7864fbd3fc9c46a491f8ad848066da22 in term 1.
I20260926 02:25:04.335934 1093 raft_consensus.cc:2454] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7864fbd3fc9c46a491f8ad848066da22 in term 1.
I20260926 02:25:04.336336 1157 leader_election.cc:302] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7864fbd3fc9c46a491f8ad848066da22, 911dd2be54dd45d594ef00d563bdc3a5; no voters:
I20260926 02:25:04.336724 1294 raft_consensus.cc:2781] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:04.337196 1294 raft_consensus.cc:686] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 1 LEADER]: Becoming Leader. State: Replica: 7864fbd3fc9c46a491f8ad848066da22, State: Running, Role: LEADER
I20260926 02:25:04.337786 1294 consensus_queue.cc:237] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } }
I20260926 02:25:04.342512 818 catalog_manager.cc:5495] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7864fbd3fc9c46a491f8ad848066da22 (127.12.244.195). New cstate: current_term: 1 leader_uuid: "7864fbd3fc9c46a491f8ad848066da22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7864fbd3fc9c46a491f8ad848066da22" member_type: VOTER last_known_addr { host: "127.12.244.195" port: 44865 } health_report { overall_health: HEALTHY } } }
I20260926 02:25:04.368753 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.363149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.12.244.195:44865)
0926 02:25:04.363343 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.363347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.363370 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:25:04.363624 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.363628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.363711 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.363845 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:04.364199 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.364984 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.366462 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.366882 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.366982 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.367640 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:04.368181 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:04.368301 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.368447 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.368524 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:25:04.377857 961 raft_consensus.cc:1261] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 1 FOLLOWER]: Refusing update from remote peer 7864fbd3fc9c46a491f8ad848066da22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:25:04.377859 1093 raft_consensus.cc:1261] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 1 FOLLOWER]: Refusing update from remote peer 7864fbd3fc9c46a491f8ad848066da22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:25:04.379073 1297 consensus_queue.cc:1035] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911dd2be54dd45d594ef00d563bdc3a5" member_type: VOTER last_known_addr { host: "127.12.244.193" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:04.379557 1294 consensus_queue.cc:1035] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee6d5cadda99473ca0acba5a0bece871" member_type: VOTER last_known_addr { host: "127.12.244.194" port: 44035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:04.390811 1300 mvcc.cc:205] Tried to move back new op lower bound from 7333435409919950848 to 7333435409771839488. Current Snapshot: MvccSnapshot[applied={T|T < 7333435409919950848}]
I20260926 02:25:04.396665 1301 mvcc.cc:205] Tried to move back new op lower bound from 7333435409919950848 to 7333435409771839488. Current Snapshot: MvccSnapshot[applied={T|T < 7333435409919950848 or (T in {7333435409919950848})}]
I20260926 02:25:04.418602 818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:25:04.418953 818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:25:04.423465 818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d9c2464d0ee48f1b03b22a4b4e3ba0a: Sending DeleteTablet for 3 replicas of tablet f2d4e3ced5f645c6a295f9277774d99b
I20260926 02:25:04.424801 1073 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d4e3ced5f645c6a295f9277774d99b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:25:04 UTC) from {username='slave'} at 127.0.0.1:34812
I20260926 02:25:04.424805 1203 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d4e3ced5f645c6a295f9277774d99b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:25:04 UTC) from {username='slave'} at 127.0.0.1:43528
I20260926 02:25:04.425803 1309 tablet_replica.cc:330] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: stopping tablet replica
I20260926 02:25:04.425837 941 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d4e3ced5f645c6a295f9277774d99b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:25:04 UTC) from {username='slave'} at 127.0.0.1:46222
I20260926 02:25:04.426368 1309 raft_consensus.cc:2229] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:25:04.427155 1309 raft_consensus.cc:2258] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:25:04.428213 1310 tablet_replica.cc:330] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: stopping tablet replica
I20260926 02:25:04.428400 1311 tablet_replica.cc:330] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: stopping tablet replica
I20260926 02:25:04.428761 1310 raft_consensus.cc:2229] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:25:04.428898 1311 raft_consensus.cc:2229] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:25:04.429363 1310 raft_consensus.cc:2258] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:25:04.429476 1311 raft_consensus.cc:2258] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:25:04.429774 1309 ts_tablet_manager.cc:1905] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:25:04.431643 1310 ts_tablet_manager.cc:1905] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:25:04.431902 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 879
I20260926 02:25:04.432884 1311 ts_tablet_manager.cc:1905] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:25:04.439404 1310 ts_tablet_manager.cc:1918] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:25:04.440559 1310 log.cc:1196] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-1/wal/wals/f2d4e3ced5f645c6a295f9277774d99b
I20260926 02:25:04.441381 1310 ts_tablet_manager.cc:1939] T f2d4e3ced5f645c6a295f9277774d99b P ee6d5cadda99473ca0acba5a0bece871: Deleting consensus metadata
I20260926 02:25:04.441504 1311 ts_tablet_manager.cc:1918] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:25:04.441797 1311 log.cc:1196] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-0/wal/wals/f2d4e3ced5f645c6a295f9277774d99b
I20260926 02:25:04.442474 1311 ts_tablet_manager.cc:1939] T f2d4e3ced5f645c6a295f9277774d99b P 911dd2be54dd45d594ef00d563bdc3a5: Deleting consensus metadata
I20260926 02:25:04.443698 803 catalog_manager.cc:4841] TS ee6d5cadda99473ca0acba5a0bece871 (127.12.244.194:44035): tablet f2d4e3ced5f645c6a295f9277774d99b (table test-table [id=54b6c86638974c30874814aae83f0c3a]) successfully deleted
I20260926 02:25:04.444526 804 catalog_manager.cc:4841] TS 911dd2be54dd45d594ef00d563bdc3a5 (127.12.244.193:40387): tablet f2d4e3ced5f645c6a295f9277774d99b (table test-table [id=54b6c86638974c30874814aae83f0c3a]) successfully deleted
I20260926 02:25:04.448632 1282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.433563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.12.244.193:40387)
0926 02:25:04.433741 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.433747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.440208 (+ 6461us) server_negotiation.cc:407] Connection header received
0926 02:25:04.440437 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.440444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.440570 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.440681 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:04.441337 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.442556 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.444479 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.445195 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.445367 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.446710 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:04.446722 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:04.446742 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:04.447623 (+ 881us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:04.447696 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:04.447722 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:04.447852 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.448058 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.448198 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:25:04.448894 1309 ts_tablet_manager.cc:1918] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:25:04.449150 1309 log.cc:1196] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Deleting WAL directory at /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790389169147784-13267-0/minicluster-data/ts-2/wal/wals/f2d4e3ced5f645c6a295f9277774d99b
I20260926 02:25:04.449522 1001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:04.449779 1309 ts_tablet_manager.cc:1939] T f2d4e3ced5f645c6a295f9277774d99b P 7864fbd3fc9c46a491f8ad848066da22: Deleting consensus metadata
I20260926 02:25:04.452184 806 catalog_manager.cc:4841] TS 7864fbd3fc9c46a491f8ad848066da22 (127.12.244.195:44865): tablet f2d4e3ced5f645c6a295f9277774d99b (table test-table [id=54b6c86638974c30874814aae83f0c3a]) successfully deleted
I20260926 02:25:04.549731 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 879
I20260926 02:25:04.575562 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1010
I20260926 02:25:04.584257 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.576483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.12.244.194:44035)
0926 02:25:04.576672 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.576676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.576695 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:25:04.576877 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.576881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.577043 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.577134 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:04.577686 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.578514 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.580723 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.581157 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.581281 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.582594 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:04.582610 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:04.582630 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:04.583329 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:04.583382 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:04.583404 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:04.583509 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.583719 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.583826 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:25:04.585194 1133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:04.703540 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1010
I20260926 02:25:04.723639 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1141
I20260926 02:25:04.732523 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:04.724482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.12.244.195:44865)
0926 02:25:04.724701 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:04.724707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:04.724730 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:25:04.724859 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:04.724866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:04.724997 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:04.725139 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:04.725539 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.726695 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.728086 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:04.728854 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:04.729023 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:04.730499 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:04.730511 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:04.730531 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:04.731519 (+ 988us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:04.731589 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:04.731619 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:04.731764 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:04.731949 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 02:25:04.732086 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:25:04.733567 1264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:04.860759 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1141
I20260926 02:25:04.881932 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 788
I20260926 02:25:04.883024 849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:04.999712 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 788
2026-09-26T02:25:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3024 ms)
[----------] 6 tests from EncryptionPolicyTest (23289 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:05 dist-test-slave-qfmw krb5kdc[1319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:05 dist-test-slave-qfmw krb5kdc[1319](info): set up 2 sockets
Sep 26 02:25:05 dist-test-slave-qfmw krb5kdc[1319](info): commencing operation
krb5kdc: starting...
W20260926 02:25:06.173962 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:06 dist-test-slave-qfmw krb5kdc[1319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:06.449237 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:06.807461 1332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:06.807775 1332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:06.807964 1332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:06.815549 1332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:06.815685 1332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:06.815793 1332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:06.815908 1332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:06.815989 1332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:06.826637 1332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:06.828109 1332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:06.829573 1332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:06.842669 1340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:06.842711 1338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:06.842607 1337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:06.844036 1332 server_base.cc:1034] running on GCE node
I20260926 02:25:06.844954 1332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:06.846558 1332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:06.847821 1332 hybrid_clock.cc:648] HybridClock initialized: now 1790389506847755 us; error 69 us; skew 500 ppm
I20260926 02:25:06.848243 1332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:06 dist-test-slave-qfmw krb5kdc[1319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:06.857802 1332 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:06.859700 1332 webserver.cc:460] Webserver started at http://127.0.0.1:42129/ using document root <none> and password file <none>
I20260926 02:25:06.860491 1332 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:06.860651 1332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:06.860992 1332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:06.864326 1332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "05ea63ff94e64f9d94e18ff295517ba3"
format_stamp: "Formatted at 2026-09-26 02:25:06 on dist-test-slave-qfmw"
server_key: "ac72fcb108e7eb285082b73619e7a71a"
server_key_iv: "4513b8f9e152b2b7a4b0719b8084f8df"
server_key_version: "encryptionkey@0"
I20260926 02:25:06.865213 1332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "05ea63ff94e64f9d94e18ff295517ba3"
format_stamp: "Formatted at 2026-09-26 02:25:06 on dist-test-slave-qfmw"
server_key: "ac72fcb108e7eb285082b73619e7a71a"
server_key_iv: "4513b8f9e152b2b7a4b0719b8084f8df"
server_key_version: "encryptionkey@0"
I20260926 02:25:06.870965 1332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:25:06.875237 1347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:06.876854 1332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:25:06.877095 1332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "05ea63ff94e64f9d94e18ff295517ba3"
format_stamp: "Formatted at 2026-09-26 02:25:06 on dist-test-slave-qfmw"
server_key: "ac72fcb108e7eb285082b73619e7a71a"
server_key_iv: "4513b8f9e152b2b7a4b0719b8084f8df"
server_key_version: "encryptionkey@0"
I20260926 02:25:06.877332 1332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:06.896600 1332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:06.901378 1332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:06.901693 1332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:06.922396 1332 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34259
I20260926 02:25:06.922420 1399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34259 every 8 connection(s)
I20260926 02:25:06.924095 1332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:06.926190 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1332
I20260926 02:25:06.926438 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:06.927246 13267 external_mini_cluster.cc:1428] Setting key 8658d69b22cdc1027aa89d1c33cd8d30
I20260926 02:25:06.928979 1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:06.944373 1400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3: Bootstrap starting.
Sep 26 02:25:06 dist-test-slave-qfmw krb5kdc[1319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:25:06.948834 1400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:06.950416 1400 log.cc:824] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:06.954556 1400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3: No bootstrap required, opened a new log
I20260926 02:25:06.956121 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:06.929531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48521 (local address 127.0.0.1:34259)
0926 02:25:06.930421 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:06.930441 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:06.930499 (+ 58us) server_negotiation.cc:407] Connection header received
0926 02:25:06.932862 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:06.932896 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:06.933555 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:06.934119 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:06.935748 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:06.937914 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:06.939689 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:06.940850 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:06.941079 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:06.945759 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:06.945771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:06.945792 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:06.945846 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:06.950300 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:06.951679 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:06.951696 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:06.951703 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:06.951921 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:06.952489 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:06.952493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:06.952496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:06.953224 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:06.953598 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:06.954153 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 02:25:06.954558 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":376,"threads_started":1}
I20260926 02:25:06.961198 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:06.964216 1400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ea63ff94e64f9d94e18ff295517ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260926 02:25:06.964706 1400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:06.964850 1400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ea63ff94e64f9d94e18ff295517ba3, State: Initialized, Role: FOLLOWER
I20260926 02:25:06.965554 1400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ea63ff94e64f9d94e18ff295517ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260926 02:25:06.965843 1400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:06.966012 1400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:06.966311 1400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:06.969246 1400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ea63ff94e64f9d94e18ff295517ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260926 02:25:06.969986 1400 leader_election.cc:302] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05ea63ff94e64f9d94e18ff295517ba3; no voters:
I20260926 02:25:06.970794 1400 leader_election.cc:288] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:06.970901 1410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:06.972229 1410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 05ea63ff94e64f9d94e18ff295517ba3, State: Running, Role: LEADER
I20260926 02:25:06.973038 1410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ea63ff94e64f9d94e18ff295517ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } }
I20260926 02:25:06.973640 1400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:06.976542 1411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05ea63ff94e64f9d94e18ff295517ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ea63ff94e64f9d94e18ff295517ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } } }
I20260926 02:25:06.977041 1411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:06.981307 1412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ea63ff94e64f9d94e18ff295517ba3. Latest consensus state: current_term: 1 leader_uuid: "05ea63ff94e64f9d94e18ff295517ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ea63ff94e64f9d94e18ff295517ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34259 } } }
I20260926 02:25:06.982590 1412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:06.986338 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:06.966425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47177 (local address 127.0.0.1:34259)
0926 02:25:06.966618 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:06.966626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:06.967130 (+ 504us) server_negotiation.cc:407] Connection header received
0926 02:25:06.967331 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:06.967337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:06.967438 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:06.967582 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:06.969045 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:06.970187 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:06.971948 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:06.972670 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:06.972826 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:06.975992 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:06.975999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:06.976004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:06.976051 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:06.980695 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:06.982379 (+ 1684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:06.982389 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:06.982396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:06.982688 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:06.983569 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:06.983577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:06.983583 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:06.984126 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:06.984547 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:06.985477 (+ 930us) server_negotiation.cc:299] Negotiation successful
0926 02:25:06.985783 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:25:06.989956 1415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:06.995393 1415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:07.005772 1415 catalog_manager.cc:1347] Generated new cluster ID: 9af3375d4fbe44b6a18d802164768e1f
I20260926 02:25:07.005941 1415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:07.020654 1415 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:07.022145 1415 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:07.037148 1415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ea63ff94e64f9d94e18ff295517ba3: Generated new TSK 0
I20260926 02:25:07.038347 1415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:07.047741 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1332
I20260926 02:25:07.050384 1395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:07.145802 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1332
W20260926 02:25:07.163863 1317 connection.cc:535] client connection to 127.0.0.1:34259 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34259): Connection reset by peer (error 104)
2026-09-26T02:25:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2133 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:07 dist-test-slave-qfmw krb5kdc[1432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:07 dist-test-slave-qfmw krb5kdc[1432](info): set up 2 sockets
Sep 26 02:25:07 dist-test-slave-qfmw krb5kdc[1432](info): commencing operation
krb5kdc: starting...
W20260926 02:25:08.261899 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:08 dist-test-slave-qfmw krb5kdc[1432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:08.524468 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:08.875874 1445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:08.876199 1445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:08.876372 1445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:08.884025 1445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:08.884155 1445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:08.884279 1445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:08.884389 1445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:08.884472 1445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:08.895056 1445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:08.896487 1445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:08.897915 1445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:08.911145 1450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:08.911031 1451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:08.911118 1453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:08.912057 1445 server_base.cc:1034] running on GCE node
I20260926 02:25:08.912983 1445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:08.914597 1445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:08.915845 1445 hybrid_clock.cc:648] HybridClock initialized: now 1790389508915795 us; error 51 us; skew 500 ppm
I20260926 02:25:08.916225 1445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:08 dist-test-slave-qfmw krb5kdc[1432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:08.919764 1445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:08.921479 1445 webserver.cc:460] Webserver started at http://127.0.0.1:44457/ using document root <none> and password file <none>
I20260926 02:25:08.922264 1445 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:08.922428 1445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:08.922770 1445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:08.926093 1445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "b69110f866d548718f54f9f8c39990cc"
format_stamp: "Formatted at 2026-09-26 02:25:08 on dist-test-slave-qfmw"
server_key: "7f78d22a9a767b57551e4ea4df3466b8"
server_key_iv: "a465ad9dae09c6f5afab949f618f1c78"
server_key_version: "encryptionkey@0"
I20260926 02:25:08.926959 1445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "b69110f866d548718f54f9f8c39990cc"
format_stamp: "Formatted at 2026-09-26 02:25:08 on dist-test-slave-qfmw"
server_key: "7f78d22a9a767b57551e4ea4df3466b8"
server_key_iv: "a465ad9dae09c6f5afab949f618f1c78"
server_key_version: "encryptionkey@0"
I20260926 02:25:08.932963 1445 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 02:25:08.937274 1460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:08.939333 1445 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 02:25:08.939579 1445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "b69110f866d548718f54f9f8c39990cc"
format_stamp: "Formatted at 2026-09-26 02:25:08 on dist-test-slave-qfmw"
server_key: "7f78d22a9a767b57551e4ea4df3466b8"
server_key_iv: "a465ad9dae09c6f5afab949f618f1c78"
server_key_version: "encryptionkey@0"
I20260926 02:25:08.939833 1445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:08.983247 1445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:08.987891 1445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:08.988147 1445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:09.008690 1445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46581
I20260926 02:25:09.008735 1512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46581 every 8 connection(s)
I20260926 02:25:09.010229 1445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:09.011526 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1445
I20260926 02:25:09.011725 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:09.012329 13267 external_mini_cluster.cc:1428] Setting key 5552f800b05c517d7f34648ef51e4c92
I20260926 02:25:09.015690 1513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:09.026609 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:09.014439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33979 (local address 127.0.0.1:46581)
0926 02:25:09.015353 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:09.015366 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:09.015409 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:25:09.017185 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:09.017209 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:09.017631 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:09.018050 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:09.018671 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:09.020117 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:09.021524 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:09.022261 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:09.022458 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:09.023957 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:09.023970 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:09.023995 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:09.024672 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:09.024769 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:09.024800 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:09.025025 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:09.025315 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 02:25:09.025515 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":382,"threads_started":1}
I20260926 02:25:09.028329 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc: Bootstrap starting.
I20260926 02:25:09.031184 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:09.032227 1513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:09.033399 1513 log.cc:824] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:09.036661 1513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc: No bootstrap required, opened a new log
I20260926 02:25:09.044088 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:09.037265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47183 (local address 127.0.0.1:46581)
0926 02:25:09.037469 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:09.037477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:09.037972 (+ 495us) server_negotiation.cc:407] Connection header received
0926 02:25:09.038205 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:09.038210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:09.038273 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:09.038380 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:09.038883 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:09.039822 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:09.041246 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:09.041764 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:09.041931 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:09.043229 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:09.043234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:09.043237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:09.043259 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:09.043317 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:09.043332 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:09.043470 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:09.043691 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:25:09.043757 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:25:09.046231 1513 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69110f866d548718f54f9f8c39990cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 02:25:09.046710 1513 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:09.046826 1513 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69110f866d548718f54f9f8c39990cc, State: Initialized, Role: FOLLOWER
I20260926 02:25:09.047531 1513 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69110f866d548718f54f9f8c39990cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 02:25:09.047804 1513 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:09.047960 1513 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:09.048156 1513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:09.050861 1513 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69110f866d548718f54f9f8c39990cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 02:25:09.051676 1513 leader_election.cc:302] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69110f866d548718f54f9f8c39990cc; no voters:
I20260926 02:25:09.052507 1513 leader_election.cc:288] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:09.052696 1523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:09.053442 1523 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [term 1 LEADER]: Becoming Leader. State: Replica: b69110f866d548718f54f9f8c39990cc, State: Running, Role: LEADER
I20260926 02:25:09.054097 1523 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69110f866d548718f54f9f8c39990cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } }
I20260926 02:25:09.054857 1513 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:09.057199 1524 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69110f866d548718f54f9f8c39990cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69110f866d548718f54f9f8c39990cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } } }
I20260926 02:25:09.057616 1524 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:09.057175 1525 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69110f866d548718f54f9f8c39990cc. Latest consensus state: current_term: 1 leader_uuid: "b69110f866d548718f54f9f8c39990cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69110f866d548718f54f9f8c39990cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46581 } } }
I20260926 02:25:09.058588 1525 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:09.061653 1529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:09.067883 1529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:09.078190 1529 catalog_manager.cc:1347] Generated new cluster ID: e505e88451784c0ebce3da3c20fa50c8
I20260926 02:25:09.078346 1529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:09.108217 1529 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:09.109706 1529 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:09.123703 1529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b69110f866d548718f54f9f8c39990cc: Generated new TSK 0
I20260926 02:25:09.124712 1529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:09.142855 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1445
W20260926 02:25:09.144181 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33979
W20260926 02:25:09.145316 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1445: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:09.145519 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1445
W20260926 02:25:09.163065 1430 connection.cc:535] client connection to 127.0.0.1:46581 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46581): Connection reset by peer (error 104)
2026-09-26T02:25:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:09 dist-test-slave-qfmw krb5kdc[1544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:09 dist-test-slave-qfmw krb5kdc[1544](info): set up 2 sockets
Sep 26 02:25:09 dist-test-slave-qfmw krb5kdc[1544](info): commencing operation
krb5kdc: starting...
W20260926 02:25:10.274144 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:10 dist-test-slave-qfmw krb5kdc[1544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:10.554365 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:10.936833 1557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:10.937232 1557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:10.937386 1557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:10.945323 1557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:10.945515 1557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:10.945626 1557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:10.945734 1557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:10.945811 1557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:10.956374 1557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:10.957829 1557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:10.959311 1557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:10.972867 1562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:10.972822 1563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:10.972783 1565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:10.973799 1557 server_base.cc:1034] running on GCE node
I20260926 02:25:10.974726 1557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:10.976426 1557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:10.977705 1557 hybrid_clock.cc:648] HybridClock initialized: now 1790389510977655 us; error 41 us; skew 500 ppm
I20260926 02:25:10.978124 1557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:10 dist-test-slave-qfmw krb5kdc[1544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:10.981719 1557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:10.983547 1557 webserver.cc:460] Webserver started at http://127.0.0.1:43553/ using document root <none> and password file <none>
I20260926 02:25:10.984304 1557 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:10.984462 1557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:10.984797 1557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:10.988158 1557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "e6460d7adbb94a8a99755c9301506c97"
format_stamp: "Formatted at 2026-09-26 02:25:10 on dist-test-slave-qfmw"
server_key: "0a24cb8b785063c270fc9821ffbc340b"
server_key_iv: "6fbe6fd95a9c72c2bbe44ddedd72912c"
server_key_version: "encryptionkey@0"
I20260926 02:25:10.989060 1557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "e6460d7adbb94a8a99755c9301506c97"
format_stamp: "Formatted at 2026-09-26 02:25:10 on dist-test-slave-qfmw"
server_key: "0a24cb8b785063c270fc9821ffbc340b"
server_key_iv: "6fbe6fd95a9c72c2bbe44ddedd72912c"
server_key_version: "encryptionkey@0"
I20260926 02:25:10.995064 1557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 02:25:10.999150 1572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:11.000634 1557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:25:11.000834 1557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "e6460d7adbb94a8a99755c9301506c97"
format_stamp: "Formatted at 2026-09-26 02:25:10 on dist-test-slave-qfmw"
server_key: "0a24cb8b785063c270fc9821ffbc340b"
server_key_iv: "6fbe6fd95a9c72c2bbe44ddedd72912c"
server_key_version: "encryptionkey@0"
I20260926 02:25:11.001065 1557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:11.007589 1557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:11.012322 1557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:11.012674 1557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:11.032848 1557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42453
I20260926 02:25:11.032902 1624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42453 every 8 connection(s)
I20260926 02:25:11.034443 1557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:11.038697 1625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:11.041260 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1557
I20260926 02:25:11.041463 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:11.042011 13267 external_mini_cluster.cc:1428] Setting key 200ee1a1527a49e85ad6b20bd5961e21
I20260926 02:25:11.050871 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:11.044194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34115 (local address 127.0.0.1:42453)
0926 02:25:11.045148 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:11.045161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:11.045209 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:25:11.046986 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:11.047014 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:11.047485 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:11.047908 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:11.049037 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:11.049044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:11.049059 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:11.049074 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:11.049145 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:11.049168 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:11.049429 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:11.049593 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 02:25:11.049818 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":451,"threads_started":1}
I20260926 02:25:11.052304 1625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97: Bootstrap starting.
I20260926 02:25:11.055059 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:11.056037 1625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:11.057193 1625 log.cc:824] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:11.060922 1625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97: No bootstrap required, opened a new log
I20260926 02:25:11.062728 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:11.059814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37183 (local address 127.0.0.1:42453)
0926 02:25:11.060054 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:11.060061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:11.060346 (+ 285us) server_negotiation.cc:407] Connection header received
0926 02:25:11.060644 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:11.060650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:11.060707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:11.060826 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:11.061995 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:11.062001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:11.062007 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:11.062030 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:11.062100 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:11.062118 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:11.062256 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:11.062367 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 02:25:11.062427 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 02:25:11.069839 1625 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6460d7adbb94a8a99755c9301506c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260926 02:25:11.070446 1625 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:11.070583 1625 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6460d7adbb94a8a99755c9301506c97, State: Initialized, Role: FOLLOWER
I20260926 02:25:11.071383 1625 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6460d7adbb94a8a99755c9301506c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260926 02:25:11.071672 1625 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:11.071856 1625 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:11.072054 1625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:11.074887 1625 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6460d7adbb94a8a99755c9301506c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260926 02:25:11.075619 1625 leader_election.cc:302] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6460d7adbb94a8a99755c9301506c97; no voters:
I20260926 02:25:11.076309 1625 leader_election.cc:288] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:11.076471 1635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:11.077193 1635 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [term 1 LEADER]: Becoming Leader. State: Replica: e6460d7adbb94a8a99755c9301506c97, State: Running, Role: LEADER
I20260926 02:25:11.077884 1635 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6460d7adbb94a8a99755c9301506c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } }
I20260926 02:25:11.078763 1625 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:11.081264 1636 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6460d7adbb94a8a99755c9301506c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6460d7adbb94a8a99755c9301506c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } } }
I20260926 02:25:11.081054 1637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6460d7adbb94a8a99755c9301506c97. Latest consensus state: current_term: 1 leader_uuid: "e6460d7adbb94a8a99755c9301506c97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6460d7adbb94a8a99755c9301506c97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42453 } } }
I20260926 02:25:11.081794 1637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:11.085960 1636 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:11.088704 1643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:11.095296 1643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:11.105638 1643 catalog_manager.cc:1347] Generated new cluster ID: 86e3423c013345bfa7d2b8f96a155f57
I20260926 02:25:11.105782 1643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:11.123910 1643 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:11.124502 1643 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:11.138114 1643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6460d7adbb94a8a99755c9301506c97: Generated new TSK 0
I20260926 02:25:11.139498 1643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:11.146596 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1557
W20260926 02:25:11.148308 1619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34115
W20260926 02:25:11.149185 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1557: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:11.149366 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1557
2026-09-26T02:25:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:11 dist-test-slave-qfmw krb5kdc[1657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:11 dist-test-slave-qfmw krb5kdc[1657](info): set up 2 sockets
Sep 26 02:25:11 dist-test-slave-qfmw krb5kdc[1657](info): commencing operation
krb5kdc: starting...
W20260926 02:25:12.274009 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:12 dist-test-slave-qfmw krb5kdc[1657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:12.547010 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:12.916198 1670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:12.916574 1670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:12.916725 1670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:12.924255 1670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:12.924439 1670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:12.924561 1670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:12.924680 1670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:12.924759 1670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:12.935242 1670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:12.936699 1670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:12.938148 1670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:12.950811 1675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:12.950848 1676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:12.950891 1678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:12.951890 1670 server_base.cc:1034] running on GCE node
I20260926 02:25:12.952805 1670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:12.954581 1670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:12.955826 1670 hybrid_clock.cc:648] HybridClock initialized: now 1790389512955778 us; error 48 us; skew 500 ppm
I20260926 02:25:12.956236 1670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:12 dist-test-slave-qfmw krb5kdc[1657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:12.959595 1670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:12.961314 1670 webserver.cc:460] Webserver started at http://127.0.0.1:32893/ using document root <none> and password file <none>
I20260926 02:25:12.962047 1670 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:12.962158 1670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:12.962423 1670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:12.965801 1670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "e32970c2d453402485c2b25e46f75424"
format_stamp: "Formatted at 2026-09-26 02:25:12 on dist-test-slave-qfmw"
server_key: "56b4c392590eead8bc9bab59816570d0"
server_key_iv: "e4c3e0acf055d5e09c2fa03958cb3a30"
server_key_version: "encryptionkey@0"
I20260926 02:25:12.966602 1670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "e32970c2d453402485c2b25e46f75424"
format_stamp: "Formatted at 2026-09-26 02:25:12 on dist-test-slave-qfmw"
server_key: "56b4c392590eead8bc9bab59816570d0"
server_key_iv: "e4c3e0acf055d5e09c2fa03958cb3a30"
server_key_version: "encryptionkey@0"
I20260926 02:25:12.972278 1670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:25:12.976184 1685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:12.977837 1670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:12.978008 1670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "e32970c2d453402485c2b25e46f75424"
format_stamp: "Formatted at 2026-09-26 02:25:12 on dist-test-slave-qfmw"
server_key: "56b4c392590eead8bc9bab59816570d0"
server_key_iv: "e4c3e0acf055d5e09c2fa03958cb3a30"
server_key_version: "encryptionkey@0"
I20260926 02:25:12.978282 1670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:12.998903 1670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:13.003716 1670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:13.004058 1670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:13.025041 1670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39909
I20260926 02:25:13.025087 1736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39909 every 8 connection(s)
I20260926 02:25:13.026620 1670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:13.030890 1737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:13.035400 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1670
I20260926 02:25:13.035610 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:13.036055 13267 external_mini_cluster.cc:1428] Setting key 7c9ee9b87324c0f296b18173ab4f5afa
I20260926 02:25:13.044734 1737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424: Bootstrap starting.
I20260926 02:25:13.048273 1737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:13.049468 1737 log.cc:824] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:25:13 dist-test-slave-qfmw krb5kdc[1657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:25:13.052942 1737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424: No bootstrap required, opened a new log
I20260926 02:25:13.061467 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:13.038282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60919 (local address 127.0.0.1:39909)
0926 02:25:13.039242 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:13.039256 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:13.039301 (+ 45us) server_negotiation.cc:407] Connection header received
0926 02:25:13.041006 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:13.041037 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:13.041528 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:13.042005 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:13.043559 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:13.044991 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:13.046405 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:13.047231 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:13.047448 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:13.052752 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:13.052763 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:13.052782 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:13.052831 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:13.056680 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:13.057561 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:13.057569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:13.057576 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:13.057794 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:13.058244 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:13.058251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:13.058259 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:13.058905 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:13.059295 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:13.059829 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 02:25:13.060342 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":451,"threads_started":1}
I20260926 02:25:13.061789 1737 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32970c2d453402485c2b25e46f75424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39909 } }
I20260926 02:25:13.062258 1737 raft_consensus.cc:374] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:13.062381 1737 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32970c2d453402485c2b25e46f75424, State: Initialized, Role: FOLLOWER
I20260926 02:25:13.063006 1737 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32970c2d453402485c2b25e46f75424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39909 } }
I20260926 02:25:13.063351 1737 raft_consensus.cc:388] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:13.063519 1737 raft_consensus.cc:482] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:13.063705 1737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:13.066028 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:13.066547 1737 raft_consensus.cc:504] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32970c2d453402485c2b25e46f75424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39909 } }
I20260926 02:25:13.067207 1737 leader_election.cc:302] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32970c2d453402485c2b25e46f75424; no voters:
I20260926 02:25:13.067891 1737 leader_election.cc:288] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:13.068064 1742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:13.070256 1737 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:13.073828 1742 raft_consensus.cc:686] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [term 1 LEADER]: Becoming Leader. State: Replica: e32970c2d453402485c2b25e46f75424, State: Running, Role: LEADER
I20260926 02:25:13.074710 1742 consensus_queue.cc:237] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32970c2d453402485c2b25e46f75424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39909 } }
I20260926 02:25:13.078178 1743 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e32970c2d453402485c2b25e46f75424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32970c2d453402485c2b25e46f75424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39909 } } }
I20260926 02:25:13.078384 1744 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e32970c2d453402485c2b25e46f75424. Latest consensus state: current_term: 1 leader_uuid: "e32970c2d453402485c2b25e46f75424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32970c2d453402485c2b25e46f75424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39909 } } }
I20260926 02:25:13.078683 1743 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:13.078687 1744 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:13.089510 1754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:13.096693 1754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:13.098017 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:13.074440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47453 (local address 127.0.0.1:39909)
0926 02:25:13.074638 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:13.074646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:13.075247 (+ 601us) server_negotiation.cc:407] Connection header received
0926 02:25:13.075596 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:13.075607 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:13.075732 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:13.075876 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:13.078766 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:13.080192 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:13.083014 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:13.084093 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:13.084333 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:13.088619 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:13.088629 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:13.088635 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:13.088697 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:13.093136 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:13.094325 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:13.094331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:13.094335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:13.094652 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:13.095295 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:13.095300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:13.095303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:13.095835 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:13.097181 (+ 1346us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:13.097344 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 02:25:13.097615 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:25:13.108472 1754 catalog_manager.cc:1347] Generated new cluster ID: 23d0b4d0a6fb43eba25ff5dd78244ba9
I20260926 02:25:13.108610 1754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:13.125375 1754 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:13.126856 1754 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:13.140434 1754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e32970c2d453402485c2b25e46f75424: Generated new TSK 0
I20260926 02:25:13.141498 1754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:13.160331 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1670
I20260926 02:25:13.161645 1732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:13.257527 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1670
2026-09-26T02:25:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2109 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:13 dist-test-slave-qfmw krb5kdc[1769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:13 dist-test-slave-qfmw krb5kdc[1769](info): set up 2 sockets
Sep 26 02:25:13 dist-test-slave-qfmw krb5kdc[1769](info): commencing operation
krb5kdc: starting...
W20260926 02:25:14.373972 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:14 dist-test-slave-qfmw krb5kdc[1769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:14.643218 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:14.998747 1782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:14.999141 1782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:14.999307 1782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:15.006866 1782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:15.007058 1782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:15.007207 1782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:15.007320 1782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:15.007400 1782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:15.017887 1782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:15.019407 1782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:15.020762 1782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:15.033394 1787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:15.033442 1790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:15.033366 1788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:15.034439 1782 server_base.cc:1034] running on GCE node
I20260926 02:25:15.035444 1782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:15.037012 1782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:15.038271 1782 hybrid_clock.cc:648] HybridClock initialized: now 1790389515038224 us; error 40 us; skew 500 ppm
I20260926 02:25:15.038810 1782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:15 dist-test-slave-qfmw krb5kdc[1769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:15.042537 1782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:15.044212 1782 webserver.cc:460] Webserver started at http://127.0.0.1:33743/ using document root <none> and password file <none>
I20260926 02:25:15.045120 1782 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:15.045324 1782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:15.045774 1782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:15.050737 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "8175ccad4dde48d187677e4a6b4643a5"
format_stamp: "Formatted at 2026-09-26 02:25:15 on dist-test-slave-qfmw"
server_key: "2c479804ad30843dea71ed085f72d2e3"
server_key_iv: "71639a9795a7755a036ee806f2de01e6"
server_key_version: "encryptionkey@0"
I20260926 02:25:15.051946 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "8175ccad4dde48d187677e4a6b4643a5"
format_stamp: "Formatted at 2026-09-26 02:25:15 on dist-test-slave-qfmw"
server_key: "2c479804ad30843dea71ed085f72d2e3"
server_key_iv: "71639a9795a7755a036ee806f2de01e6"
server_key_version: "encryptionkey@0"
I20260926 02:25:15.058367 1782 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:25:15.062531 1797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:15.064282 1782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:25:15.064505 1782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "8175ccad4dde48d187677e4a6b4643a5"
format_stamp: "Formatted at 2026-09-26 02:25:15 on dist-test-slave-qfmw"
server_key: "2c479804ad30843dea71ed085f72d2e3"
server_key_iv: "71639a9795a7755a036ee806f2de01e6"
server_key_version: "encryptionkey@0"
I20260926 02:25:15.064749 1782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:15.103194 1782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:15.107831 1782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:15.108177 1782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:15.128201 1782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39587
I20260926 02:25:15.128232 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39587 every 8 connection(s)
I20260926 02:25:15.129783 1782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:15.130887 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1782
I20260926 02:25:15.131119 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:15.131646 13267 external_mini_cluster.cc:1428] Setting key 066db22e871aae17c05bc7227558f8c9
I20260926 02:25:15.134776 1849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:15.146703 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:15.133658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48663 (local address 127.0.0.1:39587)
0926 02:25:15.134577 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:15.134591 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:15.134637 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:25:15.136535 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:15.136562 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:15.137015 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:15.137386 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:15.138091 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:15.139717 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:15.141709 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:15.142454 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:15.142622 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:15.143966 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:15.143975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:15.143991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:15.144606 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:15.144713 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:15.144744 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:15.144850 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:15.145167 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 02:25:15.145573 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":453,"threads_started":1}
I20260926 02:25:15.148185 1849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5: Bootstrap starting.
I20260926 02:25:15.151185 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:15.152137 1849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:15.153447 1849 log.cc:824] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:15.156932 1849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5: No bootstrap required, opened a new log
I20260926 02:25:15.163697 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:15.156670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59853 (local address 127.0.0.1:39587)
0926 02:25:15.156850 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:15.156856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:15.157223 (+ 367us) server_negotiation.cc:407] Connection header received
0926 02:25:15.157376 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:15.157381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:15.157489 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:15.157617 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:15.158177 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:15.159374 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:15.160786 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:15.161502 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:15.161630 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:15.162831 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:15.162837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:15.162842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:15.162865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:15.162920 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:15.162940 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:15.163042 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:15.163198 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 02:25:15.163311 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:25:15.166936 1849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175ccad4dde48d187677e4a6b4643a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39587 } }
I20260926 02:25:15.167420 1849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:15.167511 1849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8175ccad4dde48d187677e4a6b4643a5, State: Initialized, Role: FOLLOWER
I20260926 02:25:15.168042 1849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175ccad4dde48d187677e4a6b4643a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39587 } }
I20260926 02:25:15.168254 1849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:15.168355 1849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:15.168553 1849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:15.171265 1849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175ccad4dde48d187677e4a6b4643a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39587 } }
I20260926 02:25:15.171753 1849 leader_election.cc:302] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8175ccad4dde48d187677e4a6b4643a5; no voters:
I20260926 02:25:15.172317 1849 leader_election.cc:288] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:15.172493 1859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:15.173199 1859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [term 1 LEADER]: Becoming Leader. State: Replica: 8175ccad4dde48d187677e4a6b4643a5, State: Running, Role: LEADER
I20260926 02:25:15.173761 1859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175ccad4dde48d187677e4a6b4643a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39587 } }
I20260926 02:25:15.174520 1849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:15.176810 1861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8175ccad4dde48d187677e4a6b4643a5. Latest consensus state: current_term: 1 leader_uuid: "8175ccad4dde48d187677e4a6b4643a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175ccad4dde48d187677e4a6b4643a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39587 } } }
I20260926 02:25:15.177270 1861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:15.177933 1860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8175ccad4dde48d187677e4a6b4643a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175ccad4dde48d187677e4a6b4643a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39587 } } }
I20260926 02:25:15.178304 1866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:15.178309 1860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:15.184777 1866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:15.194959 1866 catalog_manager.cc:1347] Generated new cluster ID: f4c979e0963f4f5e90b7251242885657
I20260926 02:25:15.195122 1866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:15.206053 1866 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:15.207526 1866 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:15.215808 1866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8175ccad4dde48d187677e4a6b4643a5: Generated new TSK 0
I20260926 02:25:15.216780 1866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:15.229396 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1782
W20260926 02:25:15.230612 1844 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48663
W20260926 02:25:15.231609 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1782: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:15.231798 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1782
2026-09-26T02:25:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:15 dist-test-slave-qfmw krb5kdc[1880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:15 dist-test-slave-qfmw krb5kdc[1880](info): set up 2 sockets
Sep 26 02:25:15 dist-test-slave-qfmw krb5kdc[1880](info): commencing operation
krb5kdc: starting...
W20260926 02:25:16.374279 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:16 dist-test-slave-qfmw krb5kdc[1880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:16.646749 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:16.994239 1893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:16.994608 1893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:16.994782 1893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:17.002102 1893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:17.002292 1893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:17.002409 1893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:17.002533 1893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:17.002627 1893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:17.013556 1893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:17.014963 1893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:17.016381 1893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:17.028761 1898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:17.028831 1899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:17.028988 1901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:17.030064 1893 server_base.cc:1034] running on GCE node
I20260926 02:25:17.030916 1893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:17.032583 1893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:17.033849 1893 hybrid_clock.cc:648] HybridClock initialized: now 1790389517033804 us; error 50 us; skew 500 ppm
I20260926 02:25:17.034260 1893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:17 dist-test-slave-qfmw krb5kdc[1880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:17.037858 1893 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:17.039683 1893 webserver.cc:460] Webserver started at http://127.0.0.1:40361/ using document root <none> and password file <none>
I20260926 02:25:17.040436 1893 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:17.040599 1893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:17.040925 1893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:17.044288 1893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "0a462a6bf12b47859dbf392271b87ebd"
format_stamp: "Formatted at 2026-09-26 02:25:17 on dist-test-slave-qfmw"
server_key: "7cc4f52be8f6e0adbc1a55f89f6f6beb"
server_key_iv: "767abcdaa279f4a90d492c08cb7ed01a"
server_key_version: "encryptionkey@0"
I20260926 02:25:17.045140 1893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "0a462a6bf12b47859dbf392271b87ebd"
format_stamp: "Formatted at 2026-09-26 02:25:17 on dist-test-slave-qfmw"
server_key: "7cc4f52be8f6e0adbc1a55f89f6f6beb"
server_key_iv: "767abcdaa279f4a90d492c08cb7ed01a"
server_key_version: "encryptionkey@0"
I20260926 02:25:17.051028 1893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:25:17.055296 1908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:17.056949 1893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:17.057174 1893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "0a462a6bf12b47859dbf392271b87ebd"
format_stamp: "Formatted at 2026-09-26 02:25:17 on dist-test-slave-qfmw"
server_key: "7cc4f52be8f6e0adbc1a55f89f6f6beb"
server_key_iv: "767abcdaa279f4a90d492c08cb7ed01a"
server_key_version: "encryptionkey@0"
I20260926 02:25:17.057404 1893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:17.064057 1893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:17.069191 1893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:17.069541 1893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:17.089252 1893 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45767
I20260926 02:25:17.089298 1959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45767 every 8 connection(s)
I20260926 02:25:17.090751 1893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:17.092564 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 1893
I20260926 02:25:17.092760 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:17.093309 13267 external_mini_cluster.cc:1428] Setting key 56eedf01c2dcca8796307fd2b54541c1
I20260926 02:25:17.095710 1960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:17.101557 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:17.095270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32799 (local address 127.0.0.1:45767)
0926 02:25:17.096127 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:17.096139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:17.096181 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:25:17.097886 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:17.097908 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:17.098287 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:17.098713 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:17.099881 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:17.099887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:17.099903 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:17.099918 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:17.099989 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:17.100017 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:17.100268 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:17.100415 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 02:25:17.100615 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":357,"threads_started":1}
I20260926 02:25:17.105484 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:17.107539 1960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd: Bootstrap starting.
I20260926 02:25:17.111022 1960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:17.112571 1960 log.cc:824] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:17.112872 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:17.110033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51213 (local address 127.0.0.1:45767)
0926 02:25:17.110294 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:17.110304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:17.110799 (+ 495us) server_negotiation.cc:407] Connection header received
0926 02:25:17.110951 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:17.110957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:17.111034 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:17.111188 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:17.112192 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:17.112199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:17.112206 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:17.112226 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:17.112276 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:17.112302 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:17.112437 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:17.112532 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 02:25:17.112602 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:25:17.115868 1960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd: No bootstrap required, opened a new log
I20260926 02:25:17.123982 1960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a462a6bf12b47859dbf392271b87ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45767 } }
I20260926 02:25:17.124439 1960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:17.124553 1960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a462a6bf12b47859dbf392271b87ebd, State: Initialized, Role: FOLLOWER
I20260926 02:25:17.125203 1960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a462a6bf12b47859dbf392271b87ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45767 } }
I20260926 02:25:17.125447 1960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:17.125623 1960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:17.125792 1960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:17.128350 1960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a462a6bf12b47859dbf392271b87ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45767 } }
I20260926 02:25:17.128911 1960 leader_election.cc:302] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a462a6bf12b47859dbf392271b87ebd; no voters:
I20260926 02:25:17.129583 1960 leader_election.cc:288] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:17.129755 1970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:17.130465 1970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [term 1 LEADER]: Becoming Leader. State: Replica: 0a462a6bf12b47859dbf392271b87ebd, State: Running, Role: LEADER
I20260926 02:25:17.131064 1970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a462a6bf12b47859dbf392271b87ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45767 } }
I20260926 02:25:17.131845 1960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:17.134248 1971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a462a6bf12b47859dbf392271b87ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a462a6bf12b47859dbf392271b87ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45767 } } }
I20260926 02:25:17.134670 1971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:17.135185 1972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a462a6bf12b47859dbf392271b87ebd. Latest consensus state: current_term: 1 leader_uuid: "0a462a6bf12b47859dbf392271b87ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a462a6bf12b47859dbf392271b87ebd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45767 } } }
I20260926 02:25:17.135476 1972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:17.138820 1976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:17.144960 1976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:17.154552 1976 catalog_manager.cc:1347] Generated new cluster ID: 9b55d678fd2446019ef2035bfff188ba
I20260926 02:25:17.154702 1976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:17.164846 1976 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:17.165370 1976 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:17.173442 1976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a462a6bf12b47859dbf392271b87ebd: Generated new TSK 0
I20260926 02:25:17.174383 1976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:17.182729 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1893
W20260926 02:25:17.184660 1955 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32799
W20260926 02:25:17.185470 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 1893: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:17.185622 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 1893
2026-09-26T02:25:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:17 dist-test-slave-qfmw krb5kdc[1991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:17 dist-test-slave-qfmw krb5kdc[1991](info): set up 2 sockets
Sep 26 02:25:17 dist-test-slave-qfmw krb5kdc[1991](info): commencing operation
krb5kdc: starting...
W20260926 02:25:18.290001 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:18 dist-test-slave-qfmw krb5kdc[1991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:18.559847 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:18.922271 2004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:18.922617 2004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:18.922801 2004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:18.930390 2004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:18.930523 2004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:18.930650 2004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:18.930748 2004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:18.930835 2004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:18.941295 2004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:18.942783 2004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:18.944341 2004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:18.957764 2009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:18.957679 2012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:18.957712 2010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:18.958742 2004 server_base.cc:1034] running on GCE node
I20260926 02:25:18.959770 2004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:18.961383 2004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:18.962647 2004 hybrid_clock.cc:648] HybridClock initialized: now 1790389518962596 us; error 56 us; skew 500 ppm
I20260926 02:25:18.963055 2004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:18 dist-test-slave-qfmw krb5kdc[1991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:18.971993 2004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:18.973742 2004 webserver.cc:460] Webserver started at http://127.0.0.1:41359/ using document root <none> and password file <none>
I20260926 02:25:18.974516 2004 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:18.974675 2004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:18.975003 2004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:18.978240 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "502f97932a0c450ea302a6e76a542fb6"
format_stamp: "Formatted at 2026-09-26 02:25:18 on dist-test-slave-qfmw"
server_key: "56f5913e98a72eac8181eeb79ae2491c"
server_key_iv: "0676047bc430b913790770f3e2067512"
server_key_version: "encryptionkey@0"
I20260926 02:25:18.979053 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "502f97932a0c450ea302a6e76a542fb6"
format_stamp: "Formatted at 2026-09-26 02:25:18 on dist-test-slave-qfmw"
server_key: "56f5913e98a72eac8181eeb79ae2491c"
server_key_iv: "0676047bc430b913790770f3e2067512"
server_key_version: "encryptionkey@0"
I20260926 02:25:18.984575 2004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 02:25:18.988477 2019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:18.990000 2004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:18.990283 2004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "502f97932a0c450ea302a6e76a542fb6"
format_stamp: "Formatted at 2026-09-26 02:25:18 on dist-test-slave-qfmw"
server_key: "56f5913e98a72eac8181eeb79ae2491c"
server_key_iv: "0676047bc430b913790770f3e2067512"
server_key_version: "encryptionkey@0"
I20260926 02:25:18.990521 2004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:19.007689 2004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:19.012463 2004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:19.012789 2004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:19.032886 2004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40821
I20260926 02:25:19.032913 2070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40821 every 8 connection(s)
I20260926 02:25:19.034425 2004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:19.037106 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2004
I20260926 02:25:19.037310 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:19.037856 13267 external_mini_cluster.cc:1428] Setting key 7cdfbb14b28d0486ababc49db0c86336
I20260926 02:25:19.039028 2071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:25:19 dist-test-slave-qfmw krb5kdc[1991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:25:19.054044 2071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6: Bootstrap starting.
I20260926 02:25:19.058581 2071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:19.060189 2071 log.cc:824] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:19.064209 2071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6: No bootstrap required, opened a new log
I20260926 02:25:19.065613 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:19.039983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:36527 (local address 127.0.0.1:40821)
0926 02:25:19.040799 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:19.040816 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:19.040872 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:25:19.042797 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:19.042829 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:19.043478 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:19.043969 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:19.045530 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:19.047668 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:19.049174 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:19.050306 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:19.050508 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:19.055408 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:19.055422 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:19.055442 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:19.055506 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:19.059950 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:19.061086 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:19.061097 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:19.061108 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:19.061308 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:19.061970 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:19.061979 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:19.061984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:19.062725 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:19.063052 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:19.063731 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 02:25:19.064095 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":313,"threads_started":1}
I20260926 02:25:19.070393 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:19.073391 2071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f97932a0c450ea302a6e76a542fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260926 02:25:19.073822 2071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:19.073948 2071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502f97932a0c450ea302a6e76a542fb6, State: Initialized, Role: FOLLOWER
I20260926 02:25:19.074561 2071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f97932a0c450ea302a6e76a542fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260926 02:25:19.074823 2071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:19.075018 2071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:19.075297 2071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:19.078104 2071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f97932a0c450ea302a6e76a542fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260926 02:25:19.078778 2071 leader_election.cc:302] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 502f97932a0c450ea302a6e76a542fb6; no voters:
I20260926 02:25:19.079622 2071 leader_election.cc:288] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:19.079748 2081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:19.081127 2081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [term 1 LEADER]: Becoming Leader. State: Replica: 502f97932a0c450ea302a6e76a542fb6, State: Running, Role: LEADER
I20260926 02:25:19.081874 2081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f97932a0c450ea302a6e76a542fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } }
I20260926 02:25:19.082314 2071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:19.086163 2082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "502f97932a0c450ea302a6e76a542fb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f97932a0c450ea302a6e76a542fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } } }
I20260926 02:25:19.086766 2082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:19.088145 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 502f97932a0c450ea302a6e76a542fb6. Latest consensus state: current_term: 1 leader_uuid: "502f97932a0c450ea302a6e76a542fb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "502f97932a0c450ea302a6e76a542fb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40821 } } }
I20260926 02:25:19.088502 2087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:19.088510 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:19.094831 2087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:19.096951 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:19.075140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:40147 (local address 127.0.0.1:40821)
0926 02:25:19.075346 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:19.075352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:19.075804 (+ 452us) server_negotiation.cc:407] Connection header received
0926 02:25:19.075958 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:19.075967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:19.076061 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:19.076192 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:19.077650 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:19.078824 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:19.080700 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:19.081440 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:19.081581 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:19.085062 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:19.085070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:19.085074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:19.085122 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:19.090671 (+ 5549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:19.092347 (+ 1676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:19.092357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:19.092364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:19.092806 (+ 442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:19.093614 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:19.093620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:19.093623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:19.094369 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:19.096120 (+ 1751us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:19.096323 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 02:25:19.096545 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:25:19.109907 2087 catalog_manager.cc:1347] Generated new cluster ID: 4b4356d27e4b4b2eb448e4d4e9d26fd4
I20260926 02:25:19.110049 2087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:19.135982 2087 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:19.137442 2087 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:19.156939 2087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 502f97932a0c450ea302a6e76a542fb6: Generated new TSK 0
I20260926 02:25:19.158030 2087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:19.163884 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2004
I20260926 02:25:19.165664 2066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:19.260423 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2004
W20260926 02:25:19.278259 1989 connection.cc:535] client connection to 127.0.0.1:40821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40821): Connection reset by peer (error 104)
2026-09-26T02:25:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:19 dist-test-slave-qfmw krb5kdc[2104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:19 dist-test-slave-qfmw krb5kdc[2104](info): set up 2 sockets
Sep 26 02:25:19 dist-test-slave-qfmw krb5kdc[2104](info): commencing operation
krb5kdc: starting...
W20260926 02:25:20.370220 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:20 dist-test-slave-qfmw krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:20.666705 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39081
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:21.026242 2117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:21.026574 2117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:21.026743 2117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:21.034933 2117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:21.035079 2117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:21.035151 2117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:21.035265 2117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:21.035326 2117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:21.045894 2117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:21.047278 2117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:21.048597 2117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:21.061630 2122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:21.061687 2125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:21.061687 2123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:21.062914 2117 server_base.cc:1034] running on GCE node
I20260926 02:25:21.063912 2117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:21.065572 2117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:21.066821 2117 hybrid_clock.cc:648] HybridClock initialized: now 1790389521066774 us; error 47 us; skew 500 ppm
I20260926 02:25:21.067332 2117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:21 dist-test-slave-qfmw krb5kdc[2104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:21.070990 2117 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:21.072880 2117 webserver.cc:460] Webserver started at http://127.0.0.1:42289/ using document root <none> and password file <none>
I20260926 02:25:21.073661 2117 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:21.073828 2117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:21.074183 2117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:21.077673 2117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "67adf718dff8425fa8701b50c2f50bfb"
format_stamp: "Formatted at 2026-09-26 02:25:21 on dist-test-slave-qfmw"
server_key: "13e65dba92b1d2ceb75d4dfe4d739e98"
server_key_iv: "870314ae6b03ae650f85fc2519489c16"
server_key_version: "encryptionkey@0"
I20260926 02:25:21.078600 2117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "67adf718dff8425fa8701b50c2f50bfb"
format_stamp: "Formatted at 2026-09-26 02:25:21 on dist-test-slave-qfmw"
server_key: "13e65dba92b1d2ceb75d4dfe4d739e98"
server_key_iv: "870314ae6b03ae650f85fc2519489c16"
server_key_version: "encryptionkey@0"
I20260926 02:25:21.084810 2117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260926 02:25:21.088913 2132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:21.090484 2117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:25:21.090703 2117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "67adf718dff8425fa8701b50c2f50bfb"
format_stamp: "Formatted at 2026-09-26 02:25:21 on dist-test-slave-qfmw"
server_key: "13e65dba92b1d2ceb75d4dfe4d739e98"
server_key_iv: "870314ae6b03ae650f85fc2519489c16"
server_key_version: "encryptionkey@0"
I20260926 02:25:21.090935 2117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:21.107748 2117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:21.112869 2117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:21.113245 2117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:21.133966 2117 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39081
I20260926 02:25:21.133984 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39081 every 8 connection(s)
I20260926 02:25:21.135619 2117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:21.139844 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:21.144418 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2117
I20260926 02:25:21.144629 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:21.145115 13267 external_mini_cluster.cc:1428] Setting key 39cc7790b89bf8e49d7767d46759b4b2
I20260926 02:25:21.153637 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb: Bootstrap starting.
I20260926 02:25:21.157467 2185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:21.158852 2185 log.cc:824] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:21.159693 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:21.147182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:39339 (local address 127.0.0.1:39081)
0926 02:25:21.148071 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:21.148084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:21.148126 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:25:21.149845 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:21.149873 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:21.150389 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:21.150851 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:21.151495 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:21.152847 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:21.154293 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:21.155026 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:21.155320 (+ 294us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:21.156715 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:21.156724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:21.156742 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:21.157435 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:21.157536 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:21.157563 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:21.157770 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:21.158110 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 02:25:21.158344 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":361,"threads_started":1}
I20260926 02:25:21.162133 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb: No bootstrap required, opened a new log
I20260926 02:25:21.164178 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:21.173542 2185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67adf718dff8425fa8701b50c2f50bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260926 02:25:21.173949 2185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:21.174079 2185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67adf718dff8425fa8701b50c2f50bfb, State: Initialized, Role: FOLLOWER
I20260926 02:25:21.174887 2185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67adf718dff8425fa8701b50c2f50bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260926 02:25:21.175232 2185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:21.175447 2185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:21.175675 2185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:21.177390 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:21.170008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:43955 (local address 127.0.0.1:39081)
0926 02:25:21.170246 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:21.170256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:21.170523 (+ 267us) server_negotiation.cc:407] Connection header received
0926 02:25:21.170732 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:21.170739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:21.170831 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:21.170969 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:21.171670 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:21.172853 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:21.174365 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:21.175057 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:21.175235 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:21.176577 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:21.176584 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:21.176589 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:21.176612 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:21.176670 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:21.176692 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:21.176825 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:21.176976 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:25:21.177041 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:25:21.179046 2185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67adf718dff8425fa8701b50c2f50bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260926 02:25:21.179705 2185 leader_election.cc:302] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67adf718dff8425fa8701b50c2f50bfb; no voters:
I20260926 02:25:21.180347 2185 leader_election.cc:288] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:21.180579 2195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:21.181377 2195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [term 1 LEADER]: Becoming Leader. State: Replica: 67adf718dff8425fa8701b50c2f50bfb, State: Running, Role: LEADER
I20260926 02:25:21.181981 2195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67adf718dff8425fa8701b50c2f50bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } }
I20260926 02:25:21.182591 2185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:21.185353 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67adf718dff8425fa8701b50c2f50bfb. Latest consensus state: current_term: 1 leader_uuid: "67adf718dff8425fa8701b50c2f50bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67adf718dff8425fa8701b50c2f50bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } } }
I20260926 02:25:21.185823 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:21.186465 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67adf718dff8425fa8701b50c2f50bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67adf718dff8425fa8701b50c2f50bfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39081 } } }
I20260926 02:25:21.186887 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:21.192011 2200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:21.198529 2200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:21.210934 2200 catalog_manager.cc:1347] Generated new cluster ID: f85031ba693342cc9e60ef076b55c678
I20260926 02:25:21.211143 2200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:21.223310 2200 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:21.225334 2200 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:21.234445 2200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67adf718dff8425fa8701b50c2f50bfb: Generated new TSK 0
I20260926 02:25:21.235469 2200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:21.250864 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2117
W20260926 02:25:21.252656 2180 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.121:39339
W20260926 02:25:21.253535 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2117: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:21.253724 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2117
W20260926 02:25:21.271188 2102 connection.cc:535] client connection to 127.0.0.1:39081 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39081): Connection reset by peer (error 104)
2026-09-26T02:25:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:21 dist-test-slave-qfmw krb5kdc[2216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:21 dist-test-slave-qfmw krb5kdc[2216](info): set up 2 sockets
Sep 26 02:25:21 dist-test-slave-qfmw krb5kdc[2216](info): commencing operation
krb5kdc: starting...
W20260926 02:25:22.378242 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:22 dist-test-slave-qfmw krb5kdc[2216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:22.650944 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:23.010506 2229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:23.010824 2229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:23.011006 2229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:23.018646 2229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:23.018787 2229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:23.018882 2229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:23.019012 2229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:23.019095 2229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:23.030364 2229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:23.031850 2229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:23.033212 2229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:23.046574 2237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:23.046617 2235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:23.046562 2234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:23.047577 2229 server_base.cc:1034] running on GCE node
I20260926 02:25:23.048523 2229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:23.050228 2229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:23.051580 2229 hybrid_clock.cc:648] HybridClock initialized: now 1790389523051499 us; error 66 us; skew 500 ppm
I20260926 02:25:23.052023 2229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:23 dist-test-slave-qfmw krb5kdc[2216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:23.055833 2229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:23.057576 2229 webserver.cc:460] Webserver started at http://127.0.0.1:45299/ using document root <none> and password file <none>
I20260926 02:25:23.058322 2229 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:23.058488 2229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:23.058823 2229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:23.062098 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "3f46079e75c24b3d8e61cd4177d9b5a7"
format_stamp: "Formatted at 2026-09-26 02:25:23 on dist-test-slave-qfmw"
server_key: "3849a81c46062baa370ce4ae6e726eb8"
server_key_iv: "273b6fe870e86bd7c02fba9ddef7f388"
server_key_version: "encryptionkey@0"
I20260926 02:25:23.063064 2229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "3f46079e75c24b3d8e61cd4177d9b5a7"
format_stamp: "Formatted at 2026-09-26 02:25:23 on dist-test-slave-qfmw"
server_key: "3849a81c46062baa370ce4ae6e726eb8"
server_key_iv: "273b6fe870e86bd7c02fba9ddef7f388"
server_key_version: "encryptionkey@0"
I20260926 02:25:23.069011 2229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:25:23.073355 2244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:23.074939 2229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:25:23.075244 2229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "3f46079e75c24b3d8e61cd4177d9b5a7"
format_stamp: "Formatted at 2026-09-26 02:25:23 on dist-test-slave-qfmw"
server_key: "3849a81c46062baa370ce4ae6e726eb8"
server_key_iv: "273b6fe870e86bd7c02fba9ddef7f388"
server_key_version: "encryptionkey@0"
I20260926 02:25:23.075485 2229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:23.081849 2229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:23.086748 2229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:23.087235 2229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:23.108368 2229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37505
I20260926 02:25:23.108394 2296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37505 every 8 connection(s)
I20260926 02:25:23.110046 2229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:23.114249 2297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:23.118224 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2229
I20260926 02:25:23.118424 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:23.118984 13267 external_mini_cluster.cc:1428] Setting key 126382366c2c01801d26ce8444584492
I20260926 02:25:23.127833 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:23.120961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:38103 (local address 127.0.0.1:37505)
0926 02:25:23.121845 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:23.121859 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:23.121903 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:25:23.123753 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:23.123781 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:23.124208 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:23.124693 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:23.125859 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:23.125865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:23.125880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:23.125894 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:23.125954 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:23.125974 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:23.126224 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:23.126378 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 02:25:23.126585 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":370,"threads_started":1}
I20260926 02:25:23.128068 2297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7: Bootstrap starting.
I20260926 02:25:23.131796 2297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:23.132223 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:23.133106 2297 log.cc:824] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:23.136500 2297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7: No bootstrap required, opened a new log
I20260926 02:25:23.139878 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:23.137249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:35903 (local address 127.0.0.1:37505)
0926 02:25:23.137464 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:23.137471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:23.137787 (+ 316us) server_negotiation.cc:407] Connection header received
0926 02:25:23.137936 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:23.137940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:23.138014 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:23.138134 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:23.139153 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:23.139160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:23.139171 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:23.139193 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:23.139252 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:23.139274 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:23.139412 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:23.139466 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 02:25:23.139546 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:25:23.145624 2297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260926 02:25:23.146097 2297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:23.146229 2297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f46079e75c24b3d8e61cd4177d9b5a7, State: Initialized, Role: FOLLOWER
I20260926 02:25:23.146924 2297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260926 02:25:23.147219 2297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:23.147361 2297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:23.147548 2297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:23.149901 2297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260926 02:25:23.150482 2297 leader_election.cc:302] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f46079e75c24b3d8e61cd4177d9b5a7; no voters:
I20260926 02:25:23.151204 2297 leader_election.cc:288] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:23.151423 2307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:23.152752 2307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3f46079e75c24b3d8e61cd4177d9b5a7, State: Running, Role: LEADER
I20260926 02:25:23.153378 2307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260926 02:25:23.153658 2297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:23.156292 2309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f46079e75c24b3d8e61cd4177d9b5a7. Latest consensus state: current_term: 1 leader_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } } }
I20260926 02:25:23.156767 2309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:23.157172 2308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f46079e75c24b3d8e61cd4177d9b5a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } } }
I20260926 02:25:23.157387 2308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:23.160367 2314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:23.165277 2314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:23.176203 2314 catalog_manager.cc:1347] Generated new cluster ID: e0adbbc28e3449818669657d3939bfeb
I20260926 02:25:23.176373 2314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:23.204517 2314 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:23.205005 2314 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:23.218869 2314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f46079e75c24b3d8e61cd4177d9b5a7: Generated new TSK 0
I20260926 02:25:23.219852 2314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:23.226919 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2229
W20260926 02:25:23.228202 2292 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.121:38103
W20260926 02:25:23.229019 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:23.229193 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2229
2026-09-26T02:25:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:23 dist-test-slave-qfmw krb5kdc[2328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:23 dist-test-slave-qfmw krb5kdc[2328](info): set up 2 sockets
Sep 26 02:25:23 dist-test-slave-qfmw krb5kdc[2328](info): commencing operation
krb5kdc: starting...
W20260926 02:25:24.342123 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:24 dist-test-slave-qfmw krb5kdc[2328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:24.614665 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:24.981959 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:24.982335 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:24.982504 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:24.990149 2341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:24.990338 2341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:24.990459 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:24.990576 2341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:24.990661 2341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:25.001165 2341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:25.002643 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:25.004150 2341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:25.017117 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:25.017120 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:25.017120 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:25.017930 2341 server_base.cc:1034] running on GCE node
I20260926 02:25:25.018826 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:25.020465 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:25.021689 2341 hybrid_clock.cc:648] HybridClock initialized: now 1790389525021615 us; error 61 us; skew 500 ppm
I20260926 02:25:25.022116 2341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:25 dist-test-slave-qfmw krb5kdc[2328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:25.034260 2341 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:25.036180 2341 webserver.cc:460] Webserver started at http://127.0.0.1:39255/ using document root <none> and password file <none>
I20260926 02:25:25.037020 2341 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:25.037190 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:25.037536 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:25.041054 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "f973c36f5f0e4359b56a786cc3cd8ea9"
format_stamp: "Formatted at 2026-09-26 02:25:25 on dist-test-slave-qfmw"
server_key: "24010022ce60e3b8185e9be9bee7a69d"
server_key_iv: "47e3c802a0f04a8ec0e3da34e23cc2a8"
server_key_version: "encryptionkey@0"
I20260926 02:25:25.041983 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "f973c36f5f0e4359b56a786cc3cd8ea9"
format_stamp: "Formatted at 2026-09-26 02:25:25 on dist-test-slave-qfmw"
server_key: "24010022ce60e3b8185e9be9bee7a69d"
server_key_iv: "47e3c802a0f04a8ec0e3da34e23cc2a8"
server_key_version: "encryptionkey@0"
I20260926 02:25:25.048230 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 02:25:25.052789 2356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:25.054443 2341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260926 02:25:25.054754 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "f973c36f5f0e4359b56a786cc3cd8ea9"
format_stamp: "Formatted at 2026-09-26 02:25:25 on dist-test-slave-qfmw"
server_key: "24010022ce60e3b8185e9be9bee7a69d"
server_key_iv: "47e3c802a0f04a8ec0e3da34e23cc2a8"
server_key_version: "encryptionkey@0"
I20260926 02:25:25.054996 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:25.089309 2341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:25.094048 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:25.094372 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:25.114265 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46635
I20260926 02:25:25.114302 2408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46635 every 8 connection(s)
I20260926 02:25:25.115809 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:25.119747 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:25.123466 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2341
I20260926 02:25:25.123672 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:25.124221 13267 external_mini_cluster.cc:1428] Setting key 0e2b2a08e44ac9923274b1c394cd8cb7
I20260926 02:25:25.132746 2409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9: Bootstrap starting.
I20260926 02:25:25.136360 2409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:25.137602 2409 log.cc:824] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:25:25 dist-test-slave-qfmw krb5kdc[2328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 02:25:25.141253 2409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9: No bootstrap required, opened a new log
I20260926 02:25:25.150512 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:25.126315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:55953 (local address 127.0.0.1:46635)
0926 02:25:25.127208 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:25.127222 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:25.127265 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:25:25.129074 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:25.129102 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:25.129592 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:25.130051 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:25.131676 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:25.133128 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:25.134503 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:25.135310 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:25.135530 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:25.140292 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:25.140305 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:25.140329 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:25.140383 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:25.144909 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:25.146004 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:25.146019 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:25.146031 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:25.146278 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:25.146790 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:25.146796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:25.146799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:25.147700 (+ 901us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:25.148032 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:25.148490 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 02:25:25.148902 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":347,"threads_started":1}
I20260926 02:25:25.152534 2409 raft_consensus.cc:348] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46635 } }
I20260926 02:25:25.153019 2409 raft_consensus.cc:374] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:25.153147 2409 raft_consensus.cc:729] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f973c36f5f0e4359b56a786cc3cd8ea9, State: Initialized, Role: FOLLOWER
I20260926 02:25:25.153751 2409 consensus_queue.cc:260] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46635 } }
I20260926 02:25:25.154066 2409 raft_consensus.cc:388] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:25.154244 2409 raft_consensus.cc:482] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:25.154449 2409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:25.155444 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:25.157886 2409 raft_consensus.cc:504] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46635 } }
I20260926 02:25:25.158592 2409 leader_election.cc:302] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f973c36f5f0e4359b56a786cc3cd8ea9; no voters:
I20260926 02:25:25.159555 2409 leader_election.cc:288] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:25.159791 2416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:25.161059 2416 raft_consensus.cc:686] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [term 1 LEADER]: Becoming Leader. State: Replica: f973c36f5f0e4359b56a786cc3cd8ea9, State: Running, Role: LEADER
I20260926 02:25:25.161749 2416 consensus_queue.cc:237] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46635 } }
I20260926 02:25:25.162741 2409 sys_catalog.cc:564] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:25.166816 2420 sys_catalog.cc:455] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f973c36f5f0e4359b56a786cc3cd8ea9. Latest consensus state: current_term: 1 leader_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46635 } } }
I20260926 02:25:25.167387 2420 sys_catalog.cc:458] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:25.167869 2419 sys_catalog.cc:455] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f973c36f5f0e4359b56a786cc3cd8ea9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46635 } } }
I20260926 02:25:25.168610 2419 sys_catalog.cc:458] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:25.175675 2427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:25.182147 2427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:25.188421 2412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:25.161634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:46983 (local address 127.0.0.1:46635)
0926 02:25:25.161827 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:25.161833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:25.162519 (+ 686us) server_negotiation.cc:407] Connection header received
0926 02:25:25.162744 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:25.162751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:25.162855 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:25.163009 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:25.165421 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:25.167282 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:25.169796 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:25.171333 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:25.171703 (+ 370us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:25.176370 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:25.176383 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:25.176391 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:25.176448 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:25.182031 (+ 5583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:25.183718 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:25.183727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:25.183733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:25.183954 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:25.184775 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:25.184783 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:25.184789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:25.185393 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:25.187284 (+ 1891us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:25.187539 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 02:25:25.187826 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:25:25.193876 2427 catalog_manager.cc:1347] Generated new cluster ID: 4a3ca717794b4ec1be4cc4897a7f4ce3
I20260926 02:25:25.194015 2427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:25.217782 2427 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:25.219298 2427 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:25.227967 2427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f973c36f5f0e4359b56a786cc3cd8ea9: Generated new TSK 0
I20260926 02:25:25.228914 2427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:25.238296 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2341
I20260926 02:25:25.239705 2403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:25.337090 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2341
W20260926 02:25:25.356741 2326 connection.cc:535] client connection to 127.0.0.1:46635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46635): Connection reset by peer (error 104)
2026-09-26T02:25:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2117 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:25 dist-test-slave-qfmw krb5kdc[2441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:25 dist-test-slave-qfmw krb5kdc[2441](info): set up 2 sockets
Sep 26 02:25:25 dist-test-slave-qfmw krb5kdc[2441](info): commencing operation
krb5kdc: starting...
W20260926 02:25:26.470088 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:26 dist-test-slave-qfmw krb5kdc[2441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:26.738488 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:27.083740 2454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:27.084080 2454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:27.084254 2454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:27.092001 2454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:27.092150 2454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:27.092271 2454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:27.092391 2454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:27.092448 2454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:27.102866 2454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:27.104308 2454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:27.105696 2454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:27.118912 2460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:27.118899 2462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:27.119264 2459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:27.120189 2454 server_base.cc:1034] running on GCE node
I20260926 02:25:27.121003 2454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:27.122678 2454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:27.123940 2454 hybrid_clock.cc:648] HybridClock initialized: now 1790389527123870 us; error 57 us; skew 500 ppm
I20260926 02:25:27.124311 2454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:27 dist-test-slave-qfmw krb5kdc[2441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:27.127900 2454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:27.129855 2454 webserver.cc:460] Webserver started at http://127.0.0.1:46105/ using document root <none> and password file <none>
I20260926 02:25:27.130625 2454 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:27.130743 2454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:27.131039 2454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:27.134538 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "de969541ef96428c87f9d402b177bd03"
format_stamp: "Formatted at 2026-09-26 02:25:27 on dist-test-slave-qfmw"
server_key: "001ad2f4300b56598900d4433c437a00"
server_key_iv: "561c8dcd195af629cd405462219c6363"
server_key_version: "encryptionkey@0"
I20260926 02:25:27.135514 2454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "de969541ef96428c87f9d402b177bd03"
format_stamp: "Formatted at 2026-09-26 02:25:27 on dist-test-slave-qfmw"
server_key: "001ad2f4300b56598900d4433c437a00"
server_key_iv: "561c8dcd195af629cd405462219c6363"
server_key_version: "encryptionkey@0"
I20260926 02:25:27.141498 2454 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 02:25:27.145725 2469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:27.147774 2454 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:25:27.148012 2454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "de969541ef96428c87f9d402b177bd03"
format_stamp: "Formatted at 2026-09-26 02:25:27 on dist-test-slave-qfmw"
server_key: "001ad2f4300b56598900d4433c437a00"
server_key_iv: "561c8dcd195af629cd405462219c6363"
server_key_version: "encryptionkey@0"
I20260926 02:25:27.148241 2454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:27.162505 2454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:27.167531 2454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:27.167852 2454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:27.187932 2454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43055
I20260926 02:25:27.187954 2520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43055 every 8 connection(s)
I20260926 02:25:27.189586 2454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:27.193723 2521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:27.195533 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2454
I20260926 02:25:27.195725 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:27.196331 13267 external_mini_cluster.cc:1428] Setting key 2a30f8de1a217c73a32afe691669502a
I20260926 02:25:27.209213 2521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03: Bootstrap starting.
I20260926 02:25:27.211038 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:27.198354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:40161 (local address 127.0.0.1:43055)
0926 02:25:27.199289 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:27.199303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:27.199350 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:25:27.201044 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:27.201070 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:27.201568 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:27.201944 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:27.202581 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:27.204014 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:27.205990 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:27.206781 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:27.206930 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:27.208328 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:27.208337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:27.208353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:27.208946 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:27.209034 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:27.209060 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:27.209194 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:27.209701 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 02:25:27.209919 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":426,"threads_started":1}
I20260926 02:25:27.213178 2521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:27.214515 2521 log.cc:824] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:27.215739 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:27.217696 2521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03: No bootstrap required, opened a new log
I20260926 02:25:27.226912 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:27.220194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:40969 (local address 127.0.0.1:43055)
0926 02:25:27.220369 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:27.220394 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:27.220856 (+ 462us) server_negotiation.cc:407] Connection header received
0926 02:25:27.221112 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:27.221117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:27.221183 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:27.221301 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:27.221859 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:27.222654 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:27.224078 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:27.224591 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:27.224774 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:27.226031 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:27.226035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:27.226039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:27.226057 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:27.226121 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:27.226138 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:27.226277 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:27.226470 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 02:25:27.226527 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:25:27.226655 2521 raft_consensus.cc:348] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de969541ef96428c87f9d402b177bd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260926 02:25:27.227311 2521 raft_consensus.cc:374] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:27.227432 2521 raft_consensus.cc:729] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de969541ef96428c87f9d402b177bd03, State: Initialized, Role: FOLLOWER
I20260926 02:25:27.228137 2521 consensus_queue.cc:260] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de969541ef96428c87f9d402b177bd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260926 02:25:27.228408 2521 raft_consensus.cc:388] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:27.228572 2521 raft_consensus.cc:482] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:27.228775 2521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:27.231271 2521 raft_consensus.cc:504] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de969541ef96428c87f9d402b177bd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260926 02:25:27.231891 2521 leader_election.cc:302] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de969541ef96428c87f9d402b177bd03; no voters:
I20260926 02:25:27.232633 2521 leader_election.cc:288] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:27.232770 2531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:27.233532 2531 raft_consensus.cc:686] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [term 1 LEADER]: Becoming Leader. State: Replica: de969541ef96428c87f9d402b177bd03, State: Running, Role: LEADER
I20260926 02:25:27.234095 2531 consensus_queue.cc:237] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de969541ef96428c87f9d402b177bd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } }
I20260926 02:25:27.234926 2521 sys_catalog.cc:564] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:27.237900 2533 sys_catalog.cc:455] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de969541ef96428c87f9d402b177bd03. Latest consensus state: current_term: 1 leader_uuid: "de969541ef96428c87f9d402b177bd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de969541ef96428c87f9d402b177bd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } } }
I20260926 02:25:27.238937 2533 sys_catalog.cc:458] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:27.237739 2532 sys_catalog.cc:455] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de969541ef96428c87f9d402b177bd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de969541ef96428c87f9d402b177bd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43055 } } }
I20260926 02:25:27.239501 2532 sys_catalog.cc:458] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:27.248392 2541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:27.253850 2541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:27.263808 2541 catalog_manager.cc:1347] Generated new cluster ID: 8ffd3c8f1e8d4b7c9284d168fc517874
I20260926 02:25:27.263936 2541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:27.287083 2541 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:27.288666 2541 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:27.298659 2541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de969541ef96428c87f9d402b177bd03: Generated new TSK 0
I20260926 02:25:27.299953 2541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:27.311302 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2454
W20260926 02:25:27.312819 2516 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.121:40161
W20260926 02:25:27.313839 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2454: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:27.314002 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2454
W20260926 02:25:27.331535 2439 connection.cc:535] client connection to 127.0.0.1:43055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43055): Connection reset by peer (error 104)
2026-09-26T02:25:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:27 dist-test-slave-qfmw krb5kdc[2553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:27 dist-test-slave-qfmw krb5kdc[2553](info): set up 2 sockets
Sep 26 02:25:27 dist-test-slave-qfmw krb5kdc[2553](info): commencing operation
krb5kdc: starting...
W20260926 02:25:28.438176 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.014s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:28 dist-test-slave-qfmw krb5kdc[2553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 02:25:28.706310 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:29.057937 2566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:29.058286 2566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:29.058473 2566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:29.066316 2566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:29.066457 2566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:29.066545 2566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:29.066645 2566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:29.066736 2566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:29.077553 2566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:29.078994 2566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:29.080500 2566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:29.093691 2571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:29.093721 2574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:29.093668 2572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:29.094893 2566 server_base.cc:1034] running on GCE node
I20260926 02:25:29.095860 2566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:29.097455 2566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:29.098731 2566 hybrid_clock.cc:648] HybridClock initialized: now 1790389529098683 us; error 37 us; skew 500 ppm
I20260926 02:25:29.099247 2566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:29 dist-test-slave-qfmw krb5kdc[2553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:29.102747 2566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 02:25:29.104595 2566 webserver.cc:460] Webserver started at http://127.0.0.1:33953/ using document root <none> and password file <none>
I20260926 02:25:29.105412 2566 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:29.105576 2566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:29.105918 2566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:29.109287 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "b79efc25caf24f848fcf1107f8000ee9"
format_stamp: "Formatted at 2026-09-26 02:25:29 on dist-test-slave-qfmw"
server_key: "7b8f186f89323d264ad6ee98a9bd0b04"
server_key_iv: "533921aa6309c68663e10a81a7040713"
server_key_version: "encryptionkey@0"
I20260926 02:25:29.110141 2566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "b79efc25caf24f848fcf1107f8000ee9"
format_stamp: "Formatted at 2026-09-26 02:25:29 on dist-test-slave-qfmw"
server_key: "7b8f186f89323d264ad6ee98a9bd0b04"
server_key_iv: "533921aa6309c68663e10a81a7040713"
server_key_version: "encryptionkey@0"
I20260926 02:25:29.115993 2566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:25:29.120005 2581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:29.121676 2566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:29.121917 2566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "b79efc25caf24f848fcf1107f8000ee9"
format_stamp: "Formatted at 2026-09-26 02:25:29 on dist-test-slave-qfmw"
server_key: "7b8f186f89323d264ad6ee98a9bd0b04"
server_key_iv: "533921aa6309c68663e10a81a7040713"
server_key_version: "encryptionkey@0"
I20260926 02:25:29.122184 2566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:29.128609 2566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:29.133711 2566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:29.134001 2566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:29.153438 2566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46075
I20260926 02:25:29.153472 2632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46075 every 8 connection(s)
I20260926 02:25:29.155107 2566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:29.159509 2633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:29.163375 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2566
I20260926 02:25:29.163637 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:29.164106 13267 external_mini_cluster.cc:1428] Setting key 51a53245a318170c60fcc4b28397212e
I20260926 02:25:29.173990 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:29.166380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:48057 (local address 127.0.0.1:46075)
0926 02:25:29.167283 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:29.167299 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:29.167354 (+ 55us) server_negotiation.cc:407] Connection header received
0926 02:25:29.169502 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:29.169545 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:29.170129 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:29.170670 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:29.171805 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:29.171814 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:29.171833 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:29.171851 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:29.171932 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:29.171961 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:29.172242 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:29.172378 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:25:29.172598 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":366,"threads_started":1}
I20260926 02:25:29.174671 2633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9: Bootstrap starting.
I20260926 02:25:29.178470 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:29.178618 2633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:29.179874 2633 log.cc:824] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:29.183081 2633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9: No bootstrap required, opened a new log
I20260926 02:25:29.185949 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:29.183161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.121:40697 (local address 127.0.0.1:46075)
0926 02:25:29.183356 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:29.183362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:29.183778 (+ 416us) server_negotiation.cc:407] Connection header received
0926 02:25:29.183928 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:29.183937 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:29.184004 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:29.184136 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:29.185138 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:29.185144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:29.185149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:29.185173 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:29.185228 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:29.185253 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:29.185370 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:29.185537 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 02:25:29.185647 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:25:29.191926 2633 raft_consensus.cc:348] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79efc25caf24f848fcf1107f8000ee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260926 02:25:29.192373 2633 raft_consensus.cc:374] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:29.192483 2633 raft_consensus.cc:729] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79efc25caf24f848fcf1107f8000ee9, State: Initialized, Role: FOLLOWER
I20260926 02:25:29.193220 2633 consensus_queue.cc:260] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79efc25caf24f848fcf1107f8000ee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260926 02:25:29.193545 2633 raft_consensus.cc:388] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:29.193724 2633 raft_consensus.cc:482] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:29.193934 2633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:29.196296 2633 raft_consensus.cc:504] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79efc25caf24f848fcf1107f8000ee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260926 02:25:29.196816 2633 leader_election.cc:302] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b79efc25caf24f848fcf1107f8000ee9; no voters:
I20260926 02:25:29.197589 2633 leader_election.cc:288] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:29.197747 2643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:29.198491 2643 raft_consensus.cc:686] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [term 1 LEADER]: Becoming Leader. State: Replica: b79efc25caf24f848fcf1107f8000ee9, State: Running, Role: LEADER
I20260926 02:25:29.199070 2643 consensus_queue.cc:237] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79efc25caf24f848fcf1107f8000ee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } }
I20260926 02:25:29.199908 2633 sys_catalog.cc:564] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:29.202181 2644 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b79efc25caf24f848fcf1107f8000ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79efc25caf24f848fcf1107f8000ee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } } }
I20260926 02:25:29.202687 2644 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:29.202744 2645 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b79efc25caf24f848fcf1107f8000ee9. Latest consensus state: current_term: 1 leader_uuid: "b79efc25caf24f848fcf1107f8000ee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79efc25caf24f848fcf1107f8000ee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46075 } } }
I20260926 02:25:29.204061 2645 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:29.204210 2649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:29.209839 2649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:29.220155 2649 catalog_manager.cc:1347] Generated new cluster ID: 50c989ade00747ea993347243dcfa499
I20260926 02:25:29.220299 2649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:29.233006 2649 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:29.233458 2649 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:29.243650 2649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b79efc25caf24f848fcf1107f8000ee9: Generated new TSK 0
I20260926 02:25:29.244611 2649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:29.253052 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2566
W20260926 02:25:29.254312 2628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.121:48057
W20260926 02:25:29.255128 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2566: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:29.255283 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2566
2026-09-26T02:25:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:29 dist-test-slave-qfmw krb5kdc[2664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:29 dist-test-slave-qfmw krb5kdc[2664](info): set up 2 sockets
Sep 26 02:25:29 dist-test-slave-qfmw krb5kdc[2664](info): commencing operation
krb5kdc: starting...
W20260926 02:25:30.366103 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:30 dist-test-slave-qfmw krb5kdc[2664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:30.638901 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:32927
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:31.010933 2677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:31.011282 2677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:31.011379 2677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:31.018986 2677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:31.019140 2677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:31.019191 2677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:31.019246 2677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:31.019291 2677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:31.029678 2677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:32927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:31.031049 2677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:31.032603 2677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:31.045904 2682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:31.045849 2683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:31.045874 2685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:31.046494 2677 server_base.cc:1034] running on GCE node
I20260926 02:25:31.047528 2677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:31.049127 2677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:31.050490 2677 hybrid_clock.cc:648] HybridClock initialized: now 1790389531050437 us; error 70 us; skew 500 ppm
I20260926 02:25:31.050951 2677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:31 dist-test-slave-qfmw krb5kdc[2664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389531, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:31.054648 2677 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:31.056486 2677 webserver.cc:460] Webserver started at http://127.12.244.254:41069/ using document root <none> and password file <none>
I20260926 02:25:31.057267 2677 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:31.057426 2677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:31.057927 2677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:31.061228 2677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "677a40f3b42d4867ab93cb2cfe38baa6"
format_stamp: "Formatted at 2026-09-26 02:25:31 on dist-test-slave-qfmw"
server_key: "7d384e0faabc045ca047889ba356a9fd"
server_key_iv: "9faabf9da238207791b09ca602689aaa"
server_key_version: "encryptionkey@0"
I20260926 02:25:31.062067 2677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "677a40f3b42d4867ab93cb2cfe38baa6"
format_stamp: "Formatted at 2026-09-26 02:25:31 on dist-test-slave-qfmw"
server_key: "7d384e0faabc045ca047889ba356a9fd"
server_key_iv: "9faabf9da238207791b09ca602689aaa"
server_key_version: "encryptionkey@0"
I20260926 02:25:31.068476 2677 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:25:31.072577 2692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:31.074142 2677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260926 02:25:31.074383 2677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "677a40f3b42d4867ab93cb2cfe38baa6"
format_stamp: "Formatted at 2026-09-26 02:25:31 on dist-test-slave-qfmw"
server_key: "7d384e0faabc045ca047889ba356a9fd"
server_key_iv: "9faabf9da238207791b09ca602689aaa"
server_key_version: "encryptionkey@0"
I20260926 02:25:31.074626 2677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:31.093914 2677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:31.098831 2677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:31.099206 2677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:31.119304 2677 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:32927
I20260926 02:25:31.119318 2743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:32927 every 8 connection(s)
I20260926 02:25:31.120880 2677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:31.125216 2744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:31.125783 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2677
I20260926 02:25:31.125957 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:31.126516 13267 external_mini_cluster.cc:1428] Setting key 5712642580962e768a6da2b1897c83d7
I20260926 02:25:31.140964 2744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6: Bootstrap starting.
Sep 26 02:25:31 dist-test-slave-qfmw krb5kdc[2664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:31.145098 2744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:31.146334 2744 log.cc:824] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:31.149477 2744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6: No bootstrap required, opened a new log
I20260926 02:25:31.153179 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:31.128640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50821 (local address 127.12.244.254:32927)
0926 02:25:31.129573 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:31.129586 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:31.129633 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:25:31.132098 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:31.132140 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:31.132672 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:31.133070 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:31.135059 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:31.136612 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:31.138657 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:31.139600 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:31.139767 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:31.144882 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:31.144891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:31.144908 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:31.144954 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:31.148362 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:31.149278 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:31.149286 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:31.149292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:31.149477 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:31.150008 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:31.150012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:31.150014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:31.150574 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:31.150982 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:31.151668 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 02:25:31.152014 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":459,"threads_started":1}
I20260926 02:25:31.157673 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:31.158535 2744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32927 } }
I20260926 02:25:31.159003 2744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:31.159174 2744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677a40f3b42d4867ab93cb2cfe38baa6, State: Initialized, Role: FOLLOWER
I20260926 02:25:31.159797 2744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32927 } }
I20260926 02:25:31.160073 2744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:31.160225 2744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:31.160416 2744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:31.163167 2744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32927 } }
I20260926 02:25:31.163915 2744 leader_election.cc:302] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677a40f3b42d4867ab93cb2cfe38baa6; no voters:
I20260926 02:25:31.164620 2744 leader_election.cc:288] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:31.164813 2755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:31.165495 2755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [term 1 LEADER]: Becoming Leader. State: Replica: 677a40f3b42d4867ab93cb2cfe38baa6, State: Running, Role: LEADER
I20260926 02:25:31.166267 2755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32927 } }
I20260926 02:25:31.169620 2744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:31.169967 2756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677a40f3b42d4867ab93cb2cfe38baa6. Latest consensus state: current_term: 1 leader_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32927 } } }
I20260926 02:25:31.170292 2756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:31.169356 2757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a40f3b42d4867ab93cb2cfe38baa6" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 32927 } } }
I20260926 02:25:31.170907 2757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:31.184490 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:31.162421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52669 (local address 127.12.244.254:32927)
0926 02:25:31.162617 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:31.162624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:31.163239 (+ 615us) server_negotiation.cc:407] Connection header received
0926 02:25:31.163412 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:31.163418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:31.163508 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:31.163636 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:31.165751 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:31.166936 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:31.168908 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:31.169785 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:31.169926 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:31.174280 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:31.174292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:31.174300 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:31.174364 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:31.179825 (+ 5461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:31.181265 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:31.181275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:31.181281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:31.181492 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:31.182187 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:31.182195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:31.182201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:31.182719 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:31.183127 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:31.183666 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 02:25:31.183948 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:25:31.185422 2760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:31.191524 2760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:31.202253 2760 catalog_manager.cc:1347] Generated new cluster ID: 76086abe5f1d4cbeab47aba8bef64a65
I20260926 02:25:31.202436 2760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:31.220508 2760 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:31.222529 2760 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:31.232641 2760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 677a40f3b42d4867ab93cb2cfe38baa6: Generated new TSK 0
I20260926 02:25:31.233574 2760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:31.247545 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2677
I20260926 02:25:31.249109 2739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:31.342898 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2677
W20260926 02:25:31.360430 2662 connection.cc:535] client connection to 127.12.244.254:32927 recv error: Network error: failed to read from TLS socket (remote: 127.12.244.254:32927): Connection reset by peer (error 104)
2026-09-26T02:25:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2086 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:31 dist-test-slave-qfmw krb5kdc[2778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:31 dist-test-slave-qfmw krb5kdc[2778](info): set up 2 sockets
Sep 26 02:25:31 dist-test-slave-qfmw krb5kdc[2778](info): commencing operation
krb5kdc: starting...
W20260926 02:25:32.465952 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:32 dist-test-slave-qfmw krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:32.734251 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36109
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:33.099038 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:33.099447 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:33.099548 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:33.107054 2791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:33.107254 2791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:33.107308 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:33.107370 2791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:33.107412 2791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:33.117609 2791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:36109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:33.118973 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:33.120479 2791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:33.133155 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:33.133116 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:33.133260 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:33.134671 2791 server_base.cc:1034] running on GCE node
I20260926 02:25:33.135660 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:33.137315 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:33.138590 2791 hybrid_clock.cc:648] HybridClock initialized: now 1790389533138561 us; error 81 us; skew 500 ppm
I20260926 02:25:33.139014 2791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:33 dist-test-slave-qfmw krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389533, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:33.142520 2791 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:33.144342 2791 webserver.cc:460] Webserver started at http://127.12.244.254:37675/ using document root <none> and password file <none>
I20260926 02:25:33.145125 2791 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:33.145285 2791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:33.145614 2791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:33.148936 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "23ae8e53541e482f9c53f5e40126da22"
format_stamp: "Formatted at 2026-09-26 02:25:33 on dist-test-slave-qfmw"
server_key: "309944946646b1ee598107fcb4b7fc49"
server_key_iv: "4828bc5812642577a704a6f3a6f7f04c"
server_key_version: "encryptionkey@0"
I20260926 02:25:33.149792 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "23ae8e53541e482f9c53f5e40126da22"
format_stamp: "Formatted at 2026-09-26 02:25:33 on dist-test-slave-qfmw"
server_key: "309944946646b1ee598107fcb4b7fc49"
server_key_iv: "4828bc5812642577a704a6f3a6f7f04c"
server_key_version: "encryptionkey@0"
I20260926 02:25:33.155808 2791 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 02:25:33.160271 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:33.161942 2791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:33.162195 2791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "23ae8e53541e482f9c53f5e40126da22"
format_stamp: "Formatted at 2026-09-26 02:25:33 on dist-test-slave-qfmw"
server_key: "309944946646b1ee598107fcb4b7fc49"
server_key_iv: "4828bc5812642577a704a6f3a6f7f04c"
server_key_version: "encryptionkey@0"
I20260926 02:25:33.162443 2791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:33.181468 2791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:33.186448 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:33.186769 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:33.206920 2791 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:36109
I20260926 02:25:33.206933 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:36109 every 8 connection(s)
I20260926 02:25:33.208426 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:33.211555 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2791
I20260926 02:25:33.211756 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:33.212296 13267 external_mini_cluster.cc:1428] Setting key 1ab36ebe4c6c9bc473ab2dd69e9dd663
I20260926 02:25:33.212862 2858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:33.226279 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:33.214232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58285 (local address 127.12.244.254:36109)
0926 02:25:33.215027 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:33.215040 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:33.215083 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:25:33.216961 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:33.216993 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:33.217583 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:33.218131 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:33.218726 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:33.220179 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:33.221513 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:33.222216 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:33.222422 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:33.223806 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:33.223814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:33.223830 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:33.224493 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:33.224577 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:33.224607 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:33.224810 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:33.225095 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 02:25:33.225280 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":351,"threads_started":1}
I20260926 02:25:33.226841 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22: Bootstrap starting.
I20260926 02:25:33.230654 2858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:33.230674 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:33.231827 2858 log.cc:824] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:33.235042 2858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22: No bootstrap required, opened a new log
I20260926 02:25:33.242512 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:33.235276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46123 (local address 127.12.244.254:36109)
0926 02:25:33.235495 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:33.235502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:33.235914 (+ 412us) server_negotiation.cc:407] Connection header received
0926 02:25:33.236081 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:33.236086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:33.236166 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:33.236281 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:33.236820 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:33.238052 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:33.239532 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:33.240240 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:33.240386 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:33.241548 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:33.241556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:33.241568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:33.241593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:33.241670 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:33.241693 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:33.241814 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:33.242065 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 02:25:33.242123 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 02:25:33.245507 2858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ae8e53541e482f9c53f5e40126da22" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36109 } }
I20260926 02:25:33.245940 2858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:33.246065 2858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ae8e53541e482f9c53f5e40126da22, State: Initialized, Role: FOLLOWER
I20260926 02:25:33.246719 2858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ae8e53541e482f9c53f5e40126da22" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36109 } }
I20260926 02:25:33.247011 2858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:33.247213 2858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:33.247412 2858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:33.250280 2858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ae8e53541e482f9c53f5e40126da22" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36109 } }
I20260926 02:25:33.250880 2858 leader_election.cc:302] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23ae8e53541e482f9c53f5e40126da22; no voters:
I20260926 02:25:33.251585 2858 leader_election.cc:288] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:33.251760 2868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:33.252556 2868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [term 1 LEADER]: Becoming Leader. State: Replica: 23ae8e53541e482f9c53f5e40126da22, State: Running, Role: LEADER
I20260926 02:25:33.253300 2868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ae8e53541e482f9c53f5e40126da22" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36109 } }
I20260926 02:25:33.254209 2858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:33.256701 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23ae8e53541e482f9c53f5e40126da22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ae8e53541e482f9c53f5e40126da22" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36109 } } }
I20260926 02:25:33.256642 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23ae8e53541e482f9c53f5e40126da22. Latest consensus state: current_term: 1 leader_uuid: "23ae8e53541e482f9c53f5e40126da22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ae8e53541e482f9c53f5e40126da22" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 36109 } } }
I20260926 02:25:33.257102 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:33.257912 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:33.260077 2875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:33.265467 2875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:33.275790 2875 catalog_manager.cc:1347] Generated new cluster ID: d4ebe841aa974033a7ef965da005d700
I20260926 02:25:33.275929 2875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:33.303931 2875 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:33.305356 2875 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:33.317211 2875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23ae8e53541e482f9c53f5e40126da22: Generated new TSK 0
I20260926 02:25:33.318230 2875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:33.325713 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2791
W20260926 02:25:33.327857 2853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58285
W20260926 02:25:33.328688 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2791: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:33.328866 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2791
W20260926 02:25:33.346132 2776 connection.cc:535] client connection to 127.12.244.254:36109 recv error: Network error: failed to read from TLS socket (remote: 127.12.244.254:36109): Connection reset by peer (error 104)
2026-09-26T02:25:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:33 dist-test-slave-qfmw krb5kdc[2889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:33 dist-test-slave-qfmw krb5kdc[2889](info): set up 2 sockets
Sep 26 02:25:33 dist-test-slave-qfmw krb5kdc[2889](info): commencing operation
krb5kdc: starting...
W20260926 02:25:34.437994 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:34 dist-test-slave-qfmw krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:34.708361 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:43047
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:35.059927 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:35.060273 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:35.060371 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:35.067975 2902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:35.068111 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:35.068159 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:35.068214 2902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:35.068260 2902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:35.078776 2902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:43047
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:43047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:35.080225 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:35.081561 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:35.099479 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:35.099463 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:35.100272 2911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:35.100360 2902 server_base.cc:1034] running on GCE node
I20260926 02:25:35.101408 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:35.103012 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:35.104310 2902 hybrid_clock.cc:648] HybridClock initialized: now 1790389535104235 us; error 49 us; skew 500 ppm
I20260926 02:25:35.104768 2902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:35 dist-test-slave-qfmw krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389535, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:35.108085 2902 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:35.109679 2902 webserver.cc:460] Webserver started at http://127.12.244.254:35519/ using document root <none> and password file <none>
I20260926 02:25:35.110394 2902 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:35.110558 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:35.110854 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:35.114010 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "a7eba416d97d4d6b99e38db15beb33fc"
format_stamp: "Formatted at 2026-09-26 02:25:35 on dist-test-slave-qfmw"
server_key: "ec17881755443406cbab28d10d63281e"
server_key_iv: "d7fab7c7c5a6f5f0a185c7ea6b1cb618"
server_key_version: "encryptionkey@0"
I20260926 02:25:35.114809 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "a7eba416d97d4d6b99e38db15beb33fc"
format_stamp: "Formatted at 2026-09-26 02:25:35 on dist-test-slave-qfmw"
server_key: "ec17881755443406cbab28d10d63281e"
server_key_iv: "d7fab7c7c5a6f5f0a185c7ea6b1cb618"
server_key_version: "encryptionkey@0"
I20260926 02:25:35.120287 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:25:35.124080 2918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:35.125646 2902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:35.125870 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "a7eba416d97d4d6b99e38db15beb33fc"
format_stamp: "Formatted at 2026-09-26 02:25:35 on dist-test-slave-qfmw"
server_key: "ec17881755443406cbab28d10d63281e"
server_key_iv: "d7fab7c7c5a6f5f0a185c7ea6b1cb618"
server_key_version: "encryptionkey@0"
I20260926 02:25:35.126111 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:35.132511 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:35.136936 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:35.137262 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:35.157763 2902 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:43047
I20260926 02:25:35.157824 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:43047 every 8 connection(s)
I20260926 02:25:35.159346 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:35.163431 2970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:35.164973 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 2902
I20260926 02:25:35.165160 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:35.165719 13267 external_mini_cluster.cc:1428] Setting key c63da23d7f6e1e2ce18102fb27490234
I20260926 02:25:35.174175 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:35.167806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49667 (local address 127.12.244.254:43047)
0926 02:25:35.168719 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:35.168731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:35.168781 (+ 50us) server_negotiation.cc:407] Connection header received
0926 02:25:35.170453 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:35.170478 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:35.170877 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:35.171301 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:35.172502 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:35.172508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:35.172524 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:35.172538 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:35.172603 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:35.172657 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:35.172833 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:35.172937 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 02:25:35.173150 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":412,"threads_started":1}
I20260926 02:25:35.178840 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:35.179033 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc: Bootstrap starting.
I20260926 02:25:35.182734 2970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:35.184176 2970 log.cc:824] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:35.186838 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:35.183811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37701 (local address 127.12.244.254:43047)
0926 02:25:35.184033 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:35.184040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:35.184527 (+ 487us) server_negotiation.cc:407] Connection header received
0926 02:25:35.184708 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:35.184715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:35.184810 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:35.184950 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:35.186131 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:35.186137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:35.186141 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:35.186159 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:35.186206 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:35.186236 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:35.186352 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:35.186465 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 02:25:35.186542 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:25:35.187867 2970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc: No bootstrap required, opened a new log
I20260926 02:25:35.195544 2970 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7eba416d97d4d6b99e38db15beb33fc" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43047 } }
I20260926 02:25:35.195986 2970 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:35.196100 2970 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7eba416d97d4d6b99e38db15beb33fc, State: Initialized, Role: FOLLOWER
I20260926 02:25:35.196733 2970 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7eba416d97d4d6b99e38db15beb33fc" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43047 } }
I20260926 02:25:35.196976 2970 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:35.197146 2970 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:35.197338 2970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:35.199779 2970 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7eba416d97d4d6b99e38db15beb33fc" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43047 } }
I20260926 02:25:35.200328 2970 leader_election.cc:302] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7eba416d97d4d6b99e38db15beb33fc; no voters:
I20260926 02:25:35.200974 2970 leader_election.cc:288] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:35.201164 2980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:35.201907 2980 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [term 1 LEADER]: Becoming Leader. State: Replica: a7eba416d97d4d6b99e38db15beb33fc, State: Running, Role: LEADER
I20260926 02:25:35.202606 2980 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7eba416d97d4d6b99e38db15beb33fc" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43047 } }
I20260926 02:25:35.203251 2970 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:35.205703 2981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7eba416d97d4d6b99e38db15beb33fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7eba416d97d4d6b99e38db15beb33fc" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43047 } } }
I20260926 02:25:35.206138 2981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:35.206624 2982 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7eba416d97d4d6b99e38db15beb33fc. Latest consensus state: current_term: 1 leader_uuid: "a7eba416d97d4d6b99e38db15beb33fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7eba416d97d4d6b99e38db15beb33fc" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 43047 } } }
I20260926 02:25:35.206913 2982 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:35.208658 2988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:35.215313 2988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:35.225131 2988 catalog_manager.cc:1347] Generated new cluster ID: 0066b9a7568547fcb0cc0bae690d42b6
I20260926 02:25:35.225265 2988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:35.255002 2988 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:35.255525 2988 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:35.273458 2988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7eba416d97d4d6b99e38db15beb33fc: Generated new TSK 0
I20260926 02:25:35.274412 2988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:35.290982 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2902
W20260926 02:25:35.292611 2965 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49667
W20260926 02:25:35.293417 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 2902: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:35.293545 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 2902
2026-09-26T02:25:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:35 dist-test-slave-qfmw krb5kdc[3002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:35 dist-test-slave-qfmw krb5kdc[3002](info): set up 2 sockets
Sep 26 02:25:35 dist-test-slave-qfmw krb5kdc[3002](info): commencing operation
krb5kdc: starting...
W20260926 02:25:36.414076 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:36 dist-test-slave-qfmw krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:36.687579 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34755
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:46425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:37.059190 3016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:37.059654 3016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:37.059823 3016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:37.067750 3016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:37.067936 3016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:37.068043 3016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:37.068146 3016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:37.068220 3016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:37.078775 3016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:46425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:37.080243 3016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:37.081770 3016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:37.095109 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:37.095171 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:37.095086 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:37.096434 3016 server_base.cc:1034] running on GCE node
I20260926 02:25:37.097366 3016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:37.099254 3016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:37.100706 3016 hybrid_clock.cc:648] HybridClock initialized: now 1790389537100613 us; error 64 us; skew 500 ppm
I20260926 02:25:37.101177 3016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:37 dist-test-slave-qfmw krb5kdc[3002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389537, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:37.112066 3016 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:37.113956 3016 webserver.cc:460] Webserver started at http://127.12.244.254:41763/ using document root <none> and password file <none>
I20260926 02:25:37.114789 3016 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:37.114964 3016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:37.115373 3016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:37.118824 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "f07dc077064b4f47bcc21a1da5246ee0"
format_stamp: "Formatted at 2026-09-26 02:25:37 on dist-test-slave-qfmw"
server_key: "ce7999ee2336c986988581b4044fed57"
server_key_iv: "6f334525273597ee713c1f32c2834ed9"
server_key_version: "encryptionkey@0"
I20260926 02:25:37.119841 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "f07dc077064b4f47bcc21a1da5246ee0"
format_stamp: "Formatted at 2026-09-26 02:25:37 on dist-test-slave-qfmw"
server_key: "ce7999ee2336c986988581b4044fed57"
server_key_iv: "6f334525273597ee713c1f32c2834ed9"
server_key_version: "encryptionkey@0"
I20260926 02:25:37.126287 3016 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 02:25:37.130610 3031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:37.132381 3016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 02:25:37.132695 3016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "f07dc077064b4f47bcc21a1da5246ee0"
format_stamp: "Formatted at 2026-09-26 02:25:37 on dist-test-slave-qfmw"
server_key: "ce7999ee2336c986988581b4044fed57"
server_key_iv: "6f334525273597ee713c1f32c2834ed9"
server_key_version: "encryptionkey@0"
I20260926 02:25:37.132942 3016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:37.156323 3016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:37.162182 3016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:37.162528 3016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:37.183961 3016 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:34755
I20260926 02:25:37.183985 3082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:34755 every 8 connection(s)
I20260926 02:25:37.185550 3016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:37.190094 3083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:37.190889 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3016
I20260926 02:25:37.191148 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:37.191622 13267 external_mini_cluster.cc:1428] Setting key e453b3c4091ce3acb2afab9e2e65c77d
I20260926 02:25:37.205896 3083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0: Bootstrap starting.
Sep 26 02:25:37 dist-test-slave-qfmw krb5kdc[3002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:37.210108 3083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:37.211517 3083 log.cc:824] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:37.214722 3083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0: No bootstrap required, opened a new log
I20260926 02:25:37.217993 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:37.193779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54809 (local address 127.12.244.254:34755)
0926 02:25:37.194688 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:37.194701 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:37.194752 (+ 51us) server_negotiation.cc:407] Connection header received
0926 02:25:37.196775 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:37.196808 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:37.197409 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:37.197844 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:37.199840 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:37.201477 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:37.203505 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:37.204401 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:37.204576 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:37.209787 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:37.209797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:37.209816 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:37.209864 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:37.213434 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:37.214264 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:37.214271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:37.214280 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:37.214467 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:37.214892 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:37.214898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:37.214900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:37.215539 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:37.215865 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:37.216329 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 02:25:37.216842 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":416,"threads_started":1}
I20260926 02:25:37.222596 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:37.223882 3083 raft_consensus.cc:348] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07dc077064b4f47bcc21a1da5246ee0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34755 } }
I20260926 02:25:37.224341 3083 raft_consensus.cc:374] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:37.224464 3083 raft_consensus.cc:729] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07dc077064b4f47bcc21a1da5246ee0, State: Initialized, Role: FOLLOWER
I20260926 02:25:37.225069 3083 consensus_queue.cc:260] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07dc077064b4f47bcc21a1da5246ee0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34755 } }
I20260926 02:25:37.225334 3083 raft_consensus.cc:388] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:37.225536 3083 raft_consensus.cc:482] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:37.225780 3083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:37.228619 3083 raft_consensus.cc:504] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07dc077064b4f47bcc21a1da5246ee0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34755 } }
I20260926 02:25:37.229329 3083 leader_election.cc:302] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f07dc077064b4f47bcc21a1da5246ee0; no voters:
I20260926 02:25:37.230113 3083 leader_election.cc:288] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:37.230232 3093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:37.232719 3083 sys_catalog.cc:564] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:37.235469 3093 raft_consensus.cc:686] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [term 1 LEADER]: Becoming Leader. State: Replica: f07dc077064b4f47bcc21a1da5246ee0, State: Running, Role: LEADER
I20260926 02:25:37.236191 3093 consensus_queue.cc:237] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07dc077064b4f47bcc21a1da5246ee0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34755 } }
I20260926 02:25:37.239480 3094 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f07dc077064b4f47bcc21a1da5246ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07dc077064b4f47bcc21a1da5246ee0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34755 } } }
I20260926 02:25:37.239948 3094 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:37.240542 3095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f07dc077064b4f47bcc21a1da5246ee0. Latest consensus state: current_term: 1 leader_uuid: "f07dc077064b4f47bcc21a1da5246ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07dc077064b4f47bcc21a1da5246ee0" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 34755 } } }
I20260926 02:25:37.240860 3095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:37.247141 3102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:37.250605 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:37.227662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58979 (local address 127.12.244.254:34755)
0926 02:25:37.227912 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:37.227935 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:37.228504 (+ 569us) server_negotiation.cc:407] Connection header received
0926 02:25:37.228722 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:37.228728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:37.228807 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:37.228926 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:37.231225 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:37.232403 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:37.234406 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:37.235502 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:37.235679 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:37.240424 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:37.240435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:37.240444 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:37.240512 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:37.245500 (+ 4988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:37.246842 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:37.246851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:37.246858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:37.247150 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:37.247786 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:37.247791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:37.247794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:37.248263 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:37.249471 (+ 1208us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:37.249686 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 02:25:37.250042 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:25:37.255286 3102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:37.265803 3102 catalog_manager.cc:1347] Generated new cluster ID: 180447d88bbb42e2bcffa56c6201133e
I20260926 02:25:37.265970 3102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:37.292503 3102 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:37.294601 3102 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:37.304255 3102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f07dc077064b4f47bcc21a1da5246ee0: Generated new TSK 0
I20260926 02:25:37.305303 3102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:37.323572 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3016
I20260926 02:25:37.325183 3078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:37.420027 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3016
2026-09-26T02:25:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:37 dist-test-slave-qfmw krb5kdc[3115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:37 dist-test-slave-qfmw krb5kdc[3115](info): set up 2 sockets
Sep 26 02:25:37 dist-test-slave-qfmw krb5kdc[3115](info): commencing operation
krb5kdc: starting...
W20260926 02:25:38.550195 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:38 dist-test-slave-qfmw krb5kdc[3115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:38.828212 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:35809
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:35809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:39.177872 3128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:39.178270 3128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:39.178438 3128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:39.186057 3128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:39.186255 3128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:39.186365 3128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:39.186533 3128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:39.186615 3128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:39.197096 3128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:35809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:35809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:39.198560 3128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:39.200044 3128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:39.213024 3133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:39.213088 3136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:39.213091 3134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:39.214452 3128 server_base.cc:1034] running on GCE node
I20260926 02:25:39.215586 3128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:39.217270 3128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:39.218577 3128 hybrid_clock.cc:648] HybridClock initialized: now 1790389539218525 us; error 75 us; skew 500 ppm
I20260926 02:25:39.219038 3128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:39 dist-test-slave-qfmw krb5kdc[3115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389539, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:39.222687 3128 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:39.224546 3128 webserver.cc:460] Webserver started at http://127.12.244.254:34179/ using document root <none> and password file <none>
I20260926 02:25:39.225328 3128 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:39.225483 3128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:39.225824 3128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:39.229236 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "c6f88a157388412ba7881db179bd9e35"
format_stamp: "Formatted at 2026-09-26 02:25:39 on dist-test-slave-qfmw"
server_key: "199edc251369fef8d5b36f704f63576e"
server_key_iv: "7004af9a20561597b1533d059fcae9a1"
server_key_version: "encryptionkey@0"
I20260926 02:25:39.230160 3128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "c6f88a157388412ba7881db179bd9e35"
format_stamp: "Formatted at 2026-09-26 02:25:39 on dist-test-slave-qfmw"
server_key: "199edc251369fef8d5b36f704f63576e"
server_key_iv: "7004af9a20561597b1533d059fcae9a1"
server_key_version: "encryptionkey@0"
I20260926 02:25:39.236173 3128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 02:25:39.240262 3143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:39.242089 3128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260926 02:25:39.242331 3128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "c6f88a157388412ba7881db179bd9e35"
format_stamp: "Formatted at 2026-09-26 02:25:39 on dist-test-slave-qfmw"
server_key: "199edc251369fef8d5b36f704f63576e"
server_key_iv: "7004af9a20561597b1533d059fcae9a1"
server_key_version: "encryptionkey@0"
I20260926 02:25:39.242569 3128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:39.269697 3128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:39.274917 3128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:39.275261 3128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:39.295545 3128 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:35809
I20260926 02:25:39.295570 3196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:35809 every 8 connection(s)
I20260926 02:25:39.297109 3128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:39.301542 3197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:39.305516 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3128
I20260926 02:25:39.305729 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:39.306499 13267 external_mini_cluster.cc:1428] Setting key 33b4f60f3943d4d2ff99455a65497d44
I20260926 02:25:39.315827 3197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35: Bootstrap starting.
I20260926 02:25:39.319854 3197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:39.321116 3197 log.cc:824] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:39.322208 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:39.308692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59717 (local address 127.12.244.254:35809)
0926 02:25:39.309603 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:39.309622 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:39.309669 (+ 47us) server_negotiation.cc:407] Connection header received
0926 02:25:39.311519 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:39.311546 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:39.312026 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:39.312451 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:39.313146 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:39.314546 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:39.316544 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:39.317317 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:39.317513 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:39.318941 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:39.319156 (+ 215us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:39.319189 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:39.319798 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:39.319891 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:39.319943 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:39.320102 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:39.320371 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 02:25:39.320738 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":443,"threads_started":1}
I20260926 02:25:39.324677 3197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35: No bootstrap required, opened a new log
I20260926 02:25:39.326505 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:39.333434 3197 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f88a157388412ba7881db179bd9e35" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35809 } }
I20260926 02:25:39.334175 3197 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:39.334307 3197 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f88a157388412ba7881db179bd9e35, State: Initialized, Role: FOLLOWER
I20260926 02:25:39.335022 3197 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f88a157388412ba7881db179bd9e35" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35809 } }
I20260926 02:25:39.335345 3197 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:39.335506 3197 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:39.335700 3197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:39.338274 3200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:39.331089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38039 (local address 127.12.244.254:35809)
0926 02:25:39.331336 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:39.331342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:39.331925 (+ 583us) server_negotiation.cc:407] Connection header received
0926 02:25:39.332183 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:39.332193 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:39.332280 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:39.332387 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:39.332921 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:39.333835 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:39.335723 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:39.336249 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:39.336412 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:39.337544 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:39.337551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:39.337559 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:39.337579 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:39.337621 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:39.337638 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:39.337732 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:39.337852 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 02:25:39.337947 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 02:25:39.338173 3197 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f88a157388412ba7881db179bd9e35" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35809 } }
I20260926 02:25:39.338805 3197 leader_election.cc:302] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f88a157388412ba7881db179bd9e35; no voters:
I20260926 02:25:39.339499 3197 leader_election.cc:288] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:39.339671 3207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:39.340425 3207 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [term 1 LEADER]: Becoming Leader. State: Replica: c6f88a157388412ba7881db179bd9e35, State: Running, Role: LEADER
I20260926 02:25:39.340997 3207 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f88a157388412ba7881db179bd9e35" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35809 } }
I20260926 02:25:39.341751 3197 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:39.344444 3209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f88a157388412ba7881db179bd9e35. Latest consensus state: current_term: 1 leader_uuid: "c6f88a157388412ba7881db179bd9e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f88a157388412ba7881db179bd9e35" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35809 } } }
I20260926 02:25:39.344929 3209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:39.344628 3208 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f88a157388412ba7881db179bd9e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f88a157388412ba7881db179bd9e35" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 35809 } } }
I20260926 02:25:39.345063 3208 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:39.347150 3212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:39.352159 3212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:39.362519 3212 catalog_manager.cc:1347] Generated new cluster ID: f137fad7919340e89836a092fb03cdbd
I20260926 02:25:39.362689 3212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:39.379894 3212 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:39.381503 3212 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:39.390594 3212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f88a157388412ba7881db179bd9e35: Generated new TSK 0
I20260926 02:25:39.391881 3212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:39.408936 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3128
W20260926 02:25:39.410485 3192 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59717
W20260926 02:25:39.411262 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3128: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:39.411425 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3128
2026-09-26T02:25:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:39 dist-test-slave-qfmw krb5kdc[3228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:39 dist-test-slave-qfmw krb5kdc[3228](info): set up 2 sockets
Sep 26 02:25:39 dist-test-slave-qfmw krb5kdc[3228](info): commencing operation
krb5kdc: starting...
W20260926 02:25:40.526093 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:40 dist-test-slave-qfmw krb5kdc[3228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:40.812752 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45507
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:45507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:41.179410 3241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:41.179752 3241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:41.179932 3241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:41.187685 3241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:41.187829 3241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:41.187911 3241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:41.188030 3241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:41.188108 3241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:41.198750 3241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:45507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:45507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:41.200261 3241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:41.201833 3241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:41.215399 3246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:41.215531 3247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:41.215751 3249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:41.216660 3241 server_base.cc:1034] running on GCE node
I20260926 02:25:41.217552 3241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:41.219228 3241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:41.220465 3241 hybrid_clock.cc:648] HybridClock initialized: now 1790389541220404 us; error 54 us; skew 500 ppm
I20260926 02:25:41.220875 3241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:41 dist-test-slave-qfmw krb5kdc[3228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389541, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:41.224656 3241 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:41.226478 3241 webserver.cc:460] Webserver started at http://127.12.244.254:46447/ using document root <none> and password file <none>
I20260926 02:25:41.227378 3241 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:41.227507 3241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:41.227814 3241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:41.231400 3241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "59a56e7b9f46417c93560baab29ac1ae"
format_stamp: "Formatted at 2026-09-26 02:25:41 on dist-test-slave-qfmw"
server_key: "578b3aa47b9a456c1eb95602f2800755"
server_key_iv: "3b17d1753f66a85e5ddc8cdc1d1d0618"
server_key_version: "encryptionkey@0"
I20260926 02:25:41.232280 3241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "59a56e7b9f46417c93560baab29ac1ae"
format_stamp: "Formatted at 2026-09-26 02:25:41 on dist-test-slave-qfmw"
server_key: "578b3aa47b9a456c1eb95602f2800755"
server_key_iv: "3b17d1753f66a85e5ddc8cdc1d1d0618"
server_key_version: "encryptionkey@0"
I20260926 02:25:41.238570 3241 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 02:25:41.242969 3256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:41.244750 3241 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:25:41.244935 3241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "59a56e7b9f46417c93560baab29ac1ae"
format_stamp: "Formatted at 2026-09-26 02:25:41 on dist-test-slave-qfmw"
server_key: "578b3aa47b9a456c1eb95602f2800755"
server_key_iv: "3b17d1753f66a85e5ddc8cdc1d1d0618"
server_key_version: "encryptionkey@0"
I20260926 02:25:41.245146 3241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:41.251643 3241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:41.256323 3241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:41.256613 3241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:41.276552 3241 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:45507
I20260926 02:25:41.276599 3307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:45507 every 8 connection(s)
I20260926 02:25:41.278144 3241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:41.279920 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3241
I20260926 02:25:41.280103 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:41.280556 13267 external_mini_cluster.cc:1428] Setting key 7da1108e51b06f4634937c28d8aa2d7f
I20260926 02:25:41.283550 3308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:41.289214 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:41.282284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39667 (local address 127.12.244.254:45507)
0926 02:25:41.283237 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:41.283258 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:41.283585 (+ 327us) server_negotiation.cc:407] Connection header received
0926 02:25:41.285409 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:41.285433 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:41.285846 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:41.286305 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:41.287432 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:41.287438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:41.287452 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:41.287466 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:41.287542 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:41.287570 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:41.287836 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:41.287983 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 02:25:41.288172 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":378,"threads_started":1}
I20260926 02:25:41.293409 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:41.296376 3308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae: Bootstrap starting.
I20260926 02:25:41.300346 3308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:41.300643 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:41.297742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41481 (local address 127.12.244.254:45507)
0926 02:25:41.297900 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:41.297904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:41.298409 (+ 505us) server_negotiation.cc:407] Connection header received
0926 02:25:41.298619 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:41.298628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:41.298687 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:41.298792 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:41.299853 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:41.299863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:41.299867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:25:41.299882 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:41.299923 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:25:41.299938 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:41.300057 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:41.300200 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:25:41.300272 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:25:41.301671 3308 log.cc:824] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:41.304806 3308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae: No bootstrap required, opened a new log
I20260926 02:25:41.312788 3308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a56e7b9f46417c93560baab29ac1ae" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45507 } }
I20260926 02:25:41.313344 3308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:25:41.313437 3308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a56e7b9f46417c93560baab29ac1ae, State: Initialized, Role: FOLLOWER
I20260926 02:25:41.314030 3308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a56e7b9f46417c93560baab29ac1ae" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45507 } }
I20260926 02:25:41.314260 3308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:25:41.314360 3308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:25:41.314500 3308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:41.316890 3308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a56e7b9f46417c93560baab29ac1ae" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45507 } }
I20260926 02:25:41.317456 3308 leader_election.cc:302] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59a56e7b9f46417c93560baab29ac1ae; no voters:
I20260926 02:25:41.318073 3308 leader_election.cc:288] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:25:41.318269 3318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:41.319119 3318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [term 1 LEADER]: Becoming Leader. State: Replica: 59a56e7b9f46417c93560baab29ac1ae, State: Running, Role: LEADER
I20260926 02:25:41.319674 3318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a56e7b9f46417c93560baab29ac1ae" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45507 } }
I20260926 02:25:41.320690 3308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:41.322883 3319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59a56e7b9f46417c93560baab29ac1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a56e7b9f46417c93560baab29ac1ae" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45507 } } }
I20260926 02:25:41.323505 3319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:41.323000 3320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59a56e7b9f46417c93560baab29ac1ae. Latest consensus state: current_term: 1 leader_uuid: "59a56e7b9f46417c93560baab29ac1ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a56e7b9f46417c93560baab29ac1ae" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 45507 } } }
I20260926 02:25:41.323670 3320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:41.325687 3323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:25:41.332003 3323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:41.343242 3323 catalog_manager.cc:1347] Generated new cluster ID: 75788a92f2f747a4bbea0cbd1bbdc70f
I20260926 02:25:41.343403 3323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:41.368902 3323 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:41.369390 3323 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:41.378576 3323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59a56e7b9f46417c93560baab29ac1ae: Generated new TSK 0
I20260926 02:25:41.379823 3323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:41.395854 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3241
W20260926 02:25:41.397574 3303 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39667
W20260926 02:25:41.398458 13267 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3241: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:25:41.398648 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3241
2026-09-26T02:25:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1987 ms)
[----------] 18 tests from AuthTokenIssuingTest (36386 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:41 dist-test-slave-qfmw krb5kdc[3339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:41 dist-test-slave-qfmw krb5kdc[3339](info): set up 2 sockets
Sep 26 02:25:41 dist-test-slave-qfmw krb5kdc[3339](info): commencing operation
krb5kdc: starting...
W20260926 02:25:42.510257 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:42 dist-test-slave-qfmw krb5kdc[3339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:42.794301 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:44181
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:44181,127.12.244.253:34955,127.12.244.252:38149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:43.156013 3353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:25:43.156411 3353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:43.156545 3353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:43.156682 3353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:43.164400 3353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:43.164522 3353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:43.164613 3353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:43.164696 3353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:43.164808 3353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:43.175366 3353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:44181,127.12.244.253:34955,127.12.244.252:38149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:44181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:43.176822 3353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:43.178263 3353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:43.190937 3361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:43.191025 3359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:43.190975 3358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:43.191764 3353 server_base.cc:1034] running on GCE node
I20260926 02:25:43.192626 3353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:43.194056 3353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:43.195258 3353 hybrid_clock.cc:648] HybridClock initialized: now 1790389543195188 us; error 44 us; skew 500 ppm
I20260926 02:25:43.195619 3353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:43 dist-test-slave-qfmw krb5kdc[3339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:43.204367 3353 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:43.206246 3353 webserver.cc:460] Webserver started at http://127.12.244.254:39125/ using document root <none> and password file <none>
I20260926 02:25:43.207063 3353 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:43.207258 3353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:43.207609 3353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:43.210878 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "277e1c84edc046bda29dfda093e32138"
format_stamp: "Formatted at 2026-09-26 02:25:43 on dist-test-slave-qfmw"
server_key: "2c6a8fd85b1ac73a33245fda0757018b"
server_key_iv: "44410c2bd82517915da06fa7af534fb8"
server_key_version: "encryptionkey@0"
I20260926 02:25:43.211815 3353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "277e1c84edc046bda29dfda093e32138"
format_stamp: "Formatted at 2026-09-26 02:25:43 on dist-test-slave-qfmw"
server_key: "2c6a8fd85b1ac73a33245fda0757018b"
server_key_iv: "44410c2bd82517915da06fa7af534fb8"
server_key_version: "encryptionkey@0"
I20260926 02:25:43.217444 3353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 02:25:43.221216 3368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:43.222626 3353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:25:43.222815 3353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "277e1c84edc046bda29dfda093e32138"
format_stamp: "Formatted at 2026-09-26 02:25:43 on dist-test-slave-qfmw"
server_key: "2c6a8fd85b1ac73a33245fda0757018b"
server_key_iv: "44410c2bd82517915da06fa7af534fb8"
server_key_version: "encryptionkey@0"
I20260926 02:25:43.223042 3353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:43.250345 3353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:43.255225 3353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:43.255550 3353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:43.275714 3353 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:44181
I20260926 02:25:43.275736 3419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:44181 every 8 connection(s)
I20260926 02:25:43.277267 3353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:43.281387 3420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:43.282559 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3353
I20260926 02:25:43.282795 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:43.283680 13267 external_mini_cluster.cc:1428] Setting key 0640a5f27130ed10190e75f02d7d2ba1
I20260926 02:25:43.292342 3420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:43 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.244.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.253@KRBTEST.COM" created.
I20260926 02:25:43.318384 3424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.296724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.12.244.254:44181)
0926 02:25:43.298516 (+ 1792us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:43.298525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:43.298583 (+ 58us) server_negotiation.cc:407] Connection header received
0926 02:25:43.300316 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:43.300321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:43.300591 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:43.300805 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:43.302212 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.303523 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.304968 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.305563 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.305776 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.310272 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:43.310281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:43.310295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:43.310329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:43.313659 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:43.314680 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:43.314688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:43.314690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:43.314857 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:43.315366 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:43.315370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:43.315372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:43.315933 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:43.316117 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:43.316836 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 02:25:43.317181 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1569,"thread_start_us":1337,"threads_started":1}
I20260926 02:25:43.318419 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.296723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:44181 (local address 127.0.0.1:57480)
0926 02:25:43.297641 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.297692 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:43.299895 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:43.300838 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:43.300846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:43.301624 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:43.301993 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.302011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.303672 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.303679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.304814 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.304827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.305816 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.305823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.305992 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.306778 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:43.306814 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:43.310020 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:43.313801 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.313813 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.313827 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.314499 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:43.314989 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.314997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.315004 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.315256 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:43.316085 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:43.316097 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:43.316407 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 02:25:43.316693 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 02:25:43.317378 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":129,"thread_start_us":386,"threads_started":1}
I20260926 02:25:43.321534 3420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:43.322872 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.322342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42436)
0926 02:25:43.322554 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.322699 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 02:25:43.323307 3369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
I20260926 02:25:43.325177 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.324667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42446)
0926 02:25:43.324854 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.324941 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 02:25:43.325387 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:34955: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab.
Entry for principal kudu/127.12.244.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab.
I20260926 02:25:43.346853 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:34955
--webserver_interface=127.12.244.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab
--principal=kudu/127.12.244.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:44181,127.12.244.253:34955,127.12.244.252:38149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:25:43.374806 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } attempt: 1
I20260926 02:25:43.376070 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.375586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42456)
0926 02:25:43.375784 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.375921 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 02:25:43.377146 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.376761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42466)
0926 02:25:43.376917 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.377004 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:25:43.377334 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:34955: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
I20260926 02:25:43.445606 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } attempt: 2
I20260926 02:25:43.446908 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.446388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42482)
0926 02:25:43.446669 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.446762 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 02:25:43.448012 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.447626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42496)
0926 02:25:43.447784 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.447861 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:25:43.448194 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:34955: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
I20260926 02:25:43.539433 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } attempt: 3
I20260926 02:25:43.540639 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.540139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42500)
0926 02:25:43.540336 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.540443 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 02:25:43.541644 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.541293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42502)
0926 02:25:43.541430 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.541508 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 02:25:43.541829 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:34955: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
I20260926 02:25:43.685101 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } attempt: 4
I20260926 02:25:43.686630 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.686043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42508)
0926 02:25:43.686387 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.686484 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 02:25:43.687994 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.687369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42510)
0926 02:25:43.687595 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.687748 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260926 02:25:43.688259 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:34955: Network error: Client connection negotiation failed: client connection to 127.12.244.253:34955: connect: Connection refused (error 111)
W20260926 02:25:43.695636 3427 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:25:43.696026 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:43.696146 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:43.696272 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:43.704607 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:43.704782 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:43.704926 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:43.705029 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:43.705149 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:43.716286 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:44181,127.12.244.253:34955,127.12.244.252:38149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab
--principal=kudu/127.12.244.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:34955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.244.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:43.717995 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:43.719584 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:43.733314 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:43.733353 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:43.733287 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:43.734447 3427 server_base.cc:1034] running on GCE node
I20260926 02:25:43.735625 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:43.737270 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:43.738514 3427 hybrid_clock.cc:648] HybridClock initialized: now 1790389543738436 us; error 50 us; skew 500 ppm
I20260926 02:25:43.738876 3427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:43 dist-test-slave-qfmw krb5kdc[3339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:43.742635 3427 init.cc:377] Logged in from keytab as kudu/127.12.244.253@KRBTEST.COM (short username kudu)
I20260926 02:25:43.744318 3427 webserver.cc:460] Webserver started at http://127.12.244.253:42863/ using document root <none> and password file <none>
I20260926 02:25:43.745141 3427 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:43.745241 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:43.745608 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:43.749111 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data/instance:
uuid: "ef269d68357c4c23a12db7dadedb43cf"
format_stamp: "Formatted at 2026-09-26 02:25:43 on dist-test-slave-qfmw"
server_key: "1fc5d98737548d0e42a3ab12899a83ed"
server_key_iv: "7c680c5d484196fdd12448fe7169e881"
server_key_version: "encryptionkey@0"
I20260926 02:25:43.750072 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/wal/instance:
uuid: "ef269d68357c4c23a12db7dadedb43cf"
format_stamp: "Formatted at 2026-09-26 02:25:43 on dist-test-slave-qfmw"
server_key: "1fc5d98737548d0e42a3ab12899a83ed"
server_key_iv: "7c680c5d484196fdd12448fe7169e881"
server_key_version: "encryptionkey@0"
I20260926 02:25:43.755920 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:25:43.759850 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:43.761600 3427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260926 02:25:43.761813 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/wal
uuid: "ef269d68357c4c23a12db7dadedb43cf"
format_stamp: "Formatted at 2026-09-26 02:25:43 on dist-test-slave-qfmw"
server_key: "1fc5d98737548d0e42a3ab12899a83ed"
server_key_iv: "7c680c5d484196fdd12448fe7169e881"
server_key_version: "encryptionkey@0"
I20260926 02:25:43.762053 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:43.784950 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:43.790710 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:43.791028 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:43.811878 3427 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.253:34955
I20260926 02:25:43.811926 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.253:34955 every 8 connection(s)
I20260926 02:25:43.813709 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
I20260926 02:25:43.817802 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:43.823980 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3427
I20260926 02:25:43.824159 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-1/wal/instance
I20260926 02:25:43.824584 13267 external_mini_cluster.cc:1428] Setting key 35eff3ad1d7ea72468898138a3b0a9c7
I20260926 02:25:43.824672 3495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:43 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:43.847890 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.827850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.12.244.254:44181)
0926 02:25:43.828427 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:43.828432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:43.829371 (+ 939us) server_negotiation.cc:407] Connection header received
0926 02:25:43.831519 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:43.831525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:43.831622 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:43.831752 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:43.833472 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.834249 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.836034 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.836464 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.836621 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.840962 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:43.840968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:43.840971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:43.841035 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:43.843936 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:43.845092 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:43.845098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:43.845102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:43.845241 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:43.845845 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:43.845849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:43.845852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:43.846189 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:43.846363 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:43.847424 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0926 02:25:43.847611 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.12.244.252@KRBTEST.COM; defaulting to no policy
I20260926 02:25:43.849545 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.828382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:44181 (local address 127.0.0.1:57490)
0926 02:25:43.829197 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.829244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:43.831194 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:43.832040 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:43.832049 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:43.832630 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:43.833236 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.833254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.834434 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.834438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.835860 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.835873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.836724 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.836730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.836986 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.837800 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:43.837825 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:43.840658 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:43.844122 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.844136 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.844161 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.844913 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:43.845365 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.845372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.845378 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.845671 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:43.846343 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:43.846356 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:43.846929 (+ 573us) client_negotiation.cc:769] Sending connection context
0926 02:25:43.847317 (+ 388us) client_negotiation.cc:240] Negotiation successful
0926 02:25:43.847935 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":328,"threads_started":1}
I20260926 02:25:43.851670 3495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.252@KRBTEST.COM" created.
Sep 26 02:25:43 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
I20260926 02:25:43.869884 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.852657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42520)
0926 02:25:43.852825 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.852841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:43.853019 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:43.854303 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:43.854307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:43.854709 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:43.854856 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.854865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.856897 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.856901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.857821 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.857833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.859175 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.859180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.859327 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.859965 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:43.859983 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:43.862389 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:43.867418 (+ 5029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.867424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.867428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.867837 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:43.868273 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.868277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.868279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.868418 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:43.869163 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:43.869167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:43.869281 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 02:25:43.869392 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:25:43.869613 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 02:25:43.870383 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.852953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.12.244.253:34955)
0926 02:25:43.853687 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:43.853699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:43.853734 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:25:43.853797 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:43.853809 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:43.854117 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:43.854284 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:43.855049 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.856728 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.857976 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.858939 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.859122 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.862718 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:43.862739 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:43.862766 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:43.862817 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:43.867257 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:43.867978 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:43.867991 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:43.867997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:43.868147 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:43.868533 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:43.868541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:43.868546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:43.869033 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:43.869249 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:43.869578 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 02:25:43.869874 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":384,"threads_started":1}
I20260926 02:25:43.873329 3495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:43.874672 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.874090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50196)
0926 02:25:43.874313 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.874461 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 02:25:43.874987 3444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:43.876938 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.876405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50212)
0926 02:25:43.876619 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.876769 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 02:25:43.877277 3495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab.
Entry for principal kudu/127.12.244.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab.
I20260926 02:25:43.891531 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:38149
--webserver_interface=127.12.244.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab
--principal=kudu/127.12.244.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:44181,127.12.244.253:34955,127.12.244.252:38149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:25:43.926641 3495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 1
I20260926 02:25:43.928045 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.927436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50222)
0926 02:25:43.927693 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.927824 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:25:43.929258 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.928855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50224)
0926 02:25:43.929034 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.929123 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:25:43.929585 3495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:43.987527 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } attempt: 5
Sep 26 02:25:43 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
I20260926 02:25:43.997829 3495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 2
I20260926 02:25:43.998979 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.998464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50228)
0926 02:25:43.998682 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.998780 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 02:25:44.000155 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.999731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50236)
0926 02:25:43.999889 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.999982 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 02:25:44.000337 3495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:44.001803 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.988455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.12.244.253:34955)
0926 02:25:43.988636 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:43.988643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:43.988666 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:25:43.988879 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:43.988883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:43.988960 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:43.989068 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:43.989854 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.990577 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.991803 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.992237 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.992370 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.995624 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:43.995630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:43.995633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:43.995666 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:43.998588 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:43.999411 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:43.999417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:43.999420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:43.999568 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.000016 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.000021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.000025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.000682 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.000836 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.001252 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.001480 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:25:44.001818 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:43.988337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42526)
0926 02:25:43.988555 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:43.988578 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:43.988728 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:43.989101 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:43.989106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:43.989513 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:43.989693 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.989702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.990718 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.990725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.991651 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:43.991663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:43.992431 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:43.992439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:43.992629 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:43.993285 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:43.993303 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:43.995405 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:43.998721 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.998726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.998730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.999226 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:43.999703 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:43.999709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:43.999712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:43.999894 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.000805 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:44.000810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:44.000977 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 02:25:44.001142 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 02:25:44.001474 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 02:25:44.003180 3420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:44.004171 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.003786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50242)
0926 02:25:44.003943 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.004028 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 02:25:44.005043 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.004730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50252)
0926 02:25:44.004865 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.004955 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 02:25:44.005234 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:44.056509 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 1
I20260926 02:25:44.057756 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.057272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50258)
0926 02:25:44.057491 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.057588 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:25:44.058830 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.058428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50262)
0926 02:25:44.058618 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.058711 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 02:25:44.059039 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:44.091624 3495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 3
I20260926 02:25:44.093108 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.092416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50274)
0926 02:25:44.092711 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.092851 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 02:25:44.094573 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.093970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50278)
0926 02:25:44.094261 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.094381 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260926 02:25:44.094885 3495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:44.127307 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 2
I20260926 02:25:44.128578 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.127983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50284)
0926 02:25:44.128270 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.128381 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 02:25:44.129767 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.129380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50292)
0926 02:25:44.129558 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.129653 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 02:25:44.130084 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:44.236343 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 3
I20260926 02:25:44.237578 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.237025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50296)
0926 02:25:44.237236 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.237341 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 02:25:44.238183 3495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 4
I20260926 02:25:44.238987 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.238410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50306)
0926 02:25:44.238704 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.238800 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260926 02:25:44.239279 3420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
I20260926 02:25:44.239344 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.238787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50318)
0926 02:25:44.239036 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.239173 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 02:25:44.240515 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.240056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50322)
0926 02:25:44.240262 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.240360 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 02:25:44.240813 3495 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:38149: Network error: Client connection negotiation failed: client connection to 127.12.244.252:38149: connect: Connection refused (error 111)
W20260926 02:25:44.242973 3503 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:25:44.243424 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:44.243551 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:44.243682 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:44.251631 3503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:44.251737 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:44.251820 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:44.251894 3503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:44.252003 3503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:44.263248 3503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:44181,127.12.244.253:34955,127.12.244.252:38149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab
--principal=kudu/127.12.244.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:38149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.244.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:44.264751 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:44.266326 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:44.279785 3508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:44.279698 3509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:44.280161 3511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:44.280913 3503 server_base.cc:1034] running on GCE node
I20260926 02:25:44.282003 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:44.284113 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:44.285396 3503 hybrid_clock.cc:648] HybridClock initialized: now 1790389544285277 us; error 89 us; skew 500 ppm
I20260926 02:25:44.285779 3503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389544, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:44.289288 3503 init.cc:377] Logged in from keytab as kudu/127.12.244.252@KRBTEST.COM (short username kudu)
I20260926 02:25:44.290979 3503 webserver.cc:460] Webserver started at http://127.12.244.252:36241/ using document root <none> and password file <none>
I20260926 02:25:44.291781 3503 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:44.291903 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:44.292263 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:44.295562 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data/instance:
uuid: "aec0246d46fb458eb2c3a951ce8a7ed6"
format_stamp: "Formatted at 2026-09-26 02:25:44 on dist-test-slave-qfmw"
server_key: "40c3d7b349e47baf85b3c767f452fbc2"
server_key_iv: "ce66b072fe0759030e61d2e7b36ecdbe"
server_key_version: "encryptionkey@0"
I20260926 02:25:44.296470 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/wal/instance:
uuid: "aec0246d46fb458eb2c3a951ce8a7ed6"
format_stamp: "Formatted at 2026-09-26 02:25:44 on dist-test-slave-qfmw"
server_key: "40c3d7b349e47baf85b3c767f452fbc2"
server_key_iv: "ce66b072fe0759030e61d2e7b36ecdbe"
server_key_version: "encryptionkey@0"
I20260926 02:25:44.302316 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260926 02:25:44.306322 3518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:44.307873 3503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 02:25:44.308073 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/wal
uuid: "aec0246d46fb458eb2c3a951ce8a7ed6"
format_stamp: "Formatted at 2026-09-26 02:25:44 on dist-test-slave-qfmw"
server_key: "40c3d7b349e47baf85b3c767f452fbc2"
server_key_iv: "ce66b072fe0759030e61d2e7b36ecdbe"
server_key_version: "encryptionkey@0"
I20260926 02:25:44.308351 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:44.332669 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:44.337445 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:44.337805 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:44.357933 3503 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.252:38149
I20260926 02:25:44.357957 3569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.252:38149 every 8 connection(s)
I20260926 02:25:44.359624 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
I20260926 02:25:44.359917 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3503
I20260926 02:25:44.360128 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790389169147784-13267-0/minicluster-data/master-2/wal/instance
I20260926 02:25:44.360802 13267 external_mini_cluster.cc:1428] Setting key 6ae9fd9963ce5185af99ed4dde78d1e8
I20260926 02:25:44.364606 3570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:44.373893 3570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389544, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:44.387866 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.362853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.12.244.252:38149)
0926 02:25:44.363757 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.363770 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.363816 (+ 46us) server_negotiation.cc:407] Connection header received
0926 02:25:44.365657 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.365683 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.366133 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.366554 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.368627 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.370171 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.372158 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.372907 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.373069 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.378397 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.378406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.378422 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.378467 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.382685 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.383631 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.383639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.383646 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.383887 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.384293 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.384297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.384302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.385057 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.385397 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.385934 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.386394 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":403,"threads_started":1}
I20260926 02:25:44.395619 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.377571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57506 (local address 127.12.244.254:44181)
0926 02:25:44.378187 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.378192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.378784 (+ 592us) server_negotiation.cc:407] Connection header received
0926 02:25:44.378999 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.379006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.379142 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.379260 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.380939 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.382057 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.383862 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.384446 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.384601 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.388583 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.388590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.388593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.388631 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.392480 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.393546 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.393552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.393555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.393747 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.394131 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.394136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.394140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.394446 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.394644 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.395142 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.395322 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":275,"threads_started":1}
I20260926 02:25:44.395813 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.377557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:44181 (local address 127.0.0.1:57506)
0926 02:25:44.378614 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.378669 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:44.378845 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:44.379338 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:44.379347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:44.380278 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:44.380740 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.380757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.382327 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.382332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.383684 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.383701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.384714 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.384719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.384854 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.385605 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:44.385623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:44.388360 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:44.392882 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.392895 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.392901 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.393400 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.393872 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.393875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.393878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.394017 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.394610 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:44.394616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:44.394781 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 02:25:44.394952 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 02:25:44.395366 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":577,"threads_started":1}
I20260926 02:25:44.397739 3570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389544, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
I20260926 02:25:44.409708 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.391431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.12.244.254:44181)
0926 02:25:44.392278 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.392284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.392313 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:25:44.392388 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.392395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.392478 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.392592 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.393774 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.394908 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.396697 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.397381 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.397524 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.402912 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.402919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.402923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.402966 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.406510 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.407388 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.407393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.407395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.407543 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.408196 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.408200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.408202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.408500 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.408891 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.409158 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.409346 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":389,"threads_started":1}
I20260926 02:25:44.414114 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.398638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:34955 (local address 127.0.0.1:42536)
0926 02:25:44.398916 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.398938 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:44.399069 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:44.399475 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:44.399482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:44.400057 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:44.400288 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.400301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.401393 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.401399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.402399 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.402411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.403181 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.403186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.403380 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.404058 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:44.404076 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:44.406611 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:44.410744 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.410749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.410773 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.411393 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.412017 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.412020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.412023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.412292 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.413163 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:44.413168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:44.413353 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 02:25:44.413466 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 02:25:44.413851 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 02:25:44.414287 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.398957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.12.244.253:34955)
0926 02:25:44.399175 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.399179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.399196 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:25:44.399249 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.399253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.399320 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.399405 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.400511 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.401236 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.402563 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.403000 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.403175 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.406890 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.406921 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.406924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.406963 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.410580 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.411535 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.411541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.411544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.411828 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.412487 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.412538 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.412541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.413011 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.413203 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.413664 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.413927 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:25:44.415509 3570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:44.416574 3420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 4
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389544, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
I20260926 02:25:44.431742 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.416474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.12.244.252:38149)
0926 02:25:44.416682 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.416687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.416705 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:25:44.416759 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.416765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.416831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.416927 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.417701 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.418672 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.420409 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.420887 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.421026 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.424764 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.424769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.424774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.424813 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.428387 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.429377 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.429381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.429384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.429539 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.429980 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.429984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.429986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.430357 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.430514 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.430982 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.431357 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 02:25:44.431890 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.411760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42548 (local address 127.12.244.253:34955)
0926 02:25:44.412553 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.412560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.412589 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:25:44.412645 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.412649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.412707 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.412787 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.414488 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.415724 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.417555 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.418086 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.418232 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.424192 (+ 5960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.424198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.424202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.424272 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.428393 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.429394 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.429399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.429403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.429559 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.429989 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.429994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.429997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.430357 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.430633 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.431245 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.431475 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":366,"threads_started":1}
I20260926 02:25:44.432533 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.416355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50328)
0926 02:25:44.416544 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.416565 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:44.416671 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:44.416944 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:44.416948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:44.417368 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:44.417530 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.417541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.418796 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.418804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.420255 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.420267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.421044 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.421050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.421241 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.421839 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:44.421862 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:44.424585 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:44.428547 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.428552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.428556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.429250 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.429684 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.429688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.429691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.429850 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.430563 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:44.430567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:44.430741 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 02:25:44.430898 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:25:44.432124 (+ 1226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":865}
I20260926 02:25:44.435256 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.417233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50330)
0926 02:25:44.417567 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.417588 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:44.417701 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:44.418895 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:44.418902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:44.419458 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:44.419695 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.419713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.421616 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.421625 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.422588 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.422601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.423585 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.423593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.423804 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.424419 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:44.424443 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:44.427294 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:44.431520 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.431528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.431535 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.432174 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.432736 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.432742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.432746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.432949 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.434120 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:44.434126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:44.434322 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 02:25:44.434487 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 02:25:44.434810 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260926 02:25:44.435988 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.417883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.12.244.252:38149)
0926 02:25:44.418533 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.418538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.418560 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:25:44.418607 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.418611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.418689 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.418853 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.419894 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.421397 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.422739 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.423405 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.423532 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.427595 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.427600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.427604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.427675 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.431336 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.432348 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.432353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.432358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.432577 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.433107 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.433113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.433115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.433978 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.434152 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.435285 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.435530 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":526,"thread_start_us":310,"threads_started":1}
I20260926 02:25:44.437704 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:44.438827 3570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: Bootstrap starting.
I20260926 02:25:44.441677 3420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138: Bootstrap starting.
I20260926 02:25:44.442409 3570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:44.443710 3570 log.cc:824] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:44.441373 3395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "277e1c84edc046bda29dfda093e32138"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57510
I20260926 02:25:44.446384 3420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:44.446844 3570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: No bootstrap required, opened a new log
I20260926 02:25:44.447727 3420 log.cc:824] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:44.451607 3420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138: No bootstrap required, opened a new log
I20260926 02:25:44.454923 3570 raft_consensus.cc:348] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
I20260926 02:25:44.455412 3570 raft_consensus.cc:729] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec0246d46fb458eb2c3a951ce8a7ed6, State: Initialized, Role: FOLLOWER
I20260926 02:25:44.455912 3570 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
I20260926 02:25:44.457185 3581 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.457985 3570 sys_catalog.cc:564] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:44.457995 3581 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:44.459875 3420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
I20260926 02:25:44.460407 3420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277e1c84edc046bda29dfda093e32138, State: Initialized, Role: FOLLOWER
I20260926 02:25:44.461030 3420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
I20260926 02:25:44.462208 3587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.462802 3587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:44.463689 3420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:44.463792 3395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:25:44.463984 3395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:44.468662 3395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
W20260926 02:25:44.470275 3593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:25:44.470947 3593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:25:44.471973 3395 leader_election.cc:288] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [CANDIDATE]: Term 1 election: Requested vote from peers ef269d68357c4c23a12db7dadedb43cf (127.12.244.253:34955), aec0246d46fb458eb2c3a951ce8a7ed6 (127.12.244.252:38149)
I20260926 02:25:44.473191 3545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "277e1c84edc046bda29dfda093e32138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6"
I20260926 02:25:44.475236 3545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:44.479331 3545 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277e1c84edc046bda29dfda093e32138 in term 1.
I20260926 02:25:44.480368 3369 leader_election.cc:302] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277e1c84edc046bda29dfda093e32138, aec0246d46fb458eb2c3a951ce8a7ed6; no voters:
I20260926 02:25:44.480966 3587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:44.482987 3587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [term 1 LEADER]: Becoming Leader. State: Replica: 277e1c84edc046bda29dfda093e32138, State: Running, Role: LEADER
I20260926 02:25:44.483726 3587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
I20260926 02:25:44.490319 3601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277e1c84edc046bda29dfda093e32138. Latest consensus state: current_term: 1 leader_uuid: "277e1c84edc046bda29dfda093e32138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.490809 3601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:44.477164 3503 master_runner.cc:186] Error getting master registration for 127.12.244.254:44181: OK, instance_id { permanent_uuid: "277e1c84edc046bda29dfda093e32138" instance_seqno: 1790389543269947 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:25:44.492867 3503 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:25:44.496559 3604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 02:25:44.508671 3374 tablet.cc:2369] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:25:44.525858 3604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:44.540031 3495 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } attempt: 5
Sep 26 02:25:44 dist-test-slave-qfmw krb5kdc[3339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389543, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
I20260926 02:25:44.562252 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.541538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.12.244.252:38149)
0926 02:25:44.541747 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.541753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.541771 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:25:44.541827 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.541835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.541913 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.542011 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.543191 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.544321 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.546455 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.547268 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.547407 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.552878 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.552886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.552891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.552944 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.557331 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.558640 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.558647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.558652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.558918 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.559565 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.559570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.559573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.560056 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.560246 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.561550 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.561838 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:25:44.562894 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.540725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:38149 (local address 127.0.0.1:50348)
0926 02:25:44.541188 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:44.541210 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:44.541322 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:44.542242 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:44.542247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:44.542780 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:44.542969 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.542980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.544526 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.544531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.546219 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:44.546238 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.547899 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.547907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:44.548312 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.549265 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:44.549294 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:44.552551 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:44.557634 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.557640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.557644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.558410 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.559154 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:44.559160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:44.559163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:44.559401 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:44.560380 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:44.560389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:44.560596 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 02:25:44.560823 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 02:25:44.561189 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310}
I20260926 02:25:44.571496 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf: Bootstrap starting.
I20260926 02:25:44.577872 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.523443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.12.244.252:38149)
0926 02:25:44.523744 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.523753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.539457 (+ 15704us) server_negotiation.cc:407] Connection header received
0926 02:25:44.539706 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.539712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.539797 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.539875 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.545080 (+ 5205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.546413 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.563473 (+ 17060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.564213 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.564310 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.568215 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.568222 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.568227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.568273 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.572589 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.574240 (+ 1651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.574248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.574253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.574534 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.575122 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.575128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.575131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.575727 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.575991 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.576903 (+ 912us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.577350 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 02:25:44.587486 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:44.589075 3495 log.cc:824] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf: Log is configured to *not* fsync() on all Append() calls
W20260926 02:25:44.590224 3524 tablet.cc:2369] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:25:44.603229 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf: No bootstrap required, opened a new log
I20260926 02:25:44.610297 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.517322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.12.244.254:44181)
0926 02:25:44.517551 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.517559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.536007 (+ 18448us) server_negotiation.cc:407] Connection header received
0926 02:25:44.536415 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.536421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.536517 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.536618 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.542143 (+ 5525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.543282 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.563232 (+ 19950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.575353 (+ 12121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.575539 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.582598 (+ 7059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.582608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.582614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.582685 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.602616 (+ 19931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.603996 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.604007 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.604013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.604245 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.605131 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.605139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.605143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.605690 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.606013 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.609518 (+ 3505us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.609861 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:25:44.622742 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:44.509908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.12.244.253:34955)
0926 02:25:44.510192 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.510202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.522514 (+ 12312us) server_negotiation.cc:407] Connection header received
0926 02:25:44.525759 (+ 3245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.525767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.525874 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.525986 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.538696 (+ 12710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.539810 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.560716 (+ 20906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:44.561427 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:44.561530 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:44.593044 (+ 31514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:44.593052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:44.593056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:44.593123 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:44.597841 (+ 4718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.619310 (+ 21469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.619319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.619323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.619705 (+ 382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:44.620491 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:44.620498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:44.620505 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:44.620956 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:44.621257 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:44.621984 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 02:25:44.622333 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:25:44.624830 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
I20260926 02:25:44.625723 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef269d68357c4c23a12db7dadedb43cf, State: Initialized, Role: FOLLOWER
I20260926 02:25:44.626415 3545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [term 1 FOLLOWER]: Refusing update from remote peer 277e1c84edc046bda29dfda093e32138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:25:44.626797 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } }
I20260926 02:25:44.628110 3601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:44.651934 3581 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277e1c84edc046bda29dfda093e32138. Latest consensus state: current_term: 1 leader_uuid: "277e1c84edc046bda29dfda093e32138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.652442 3581 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:44.657828 3621 mvcc.cc:205] Tried to move back new op lower bound from 7333435574742106112 to 7333435574212378624. Current Snapshot: MvccSnapshot[applied={T|T < 7333435574742106112}]
I20260926 02:25:44.657786 3619 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.658465 3619 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:44.661074 3469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:44.665063 3581 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "277e1c84edc046bda29dfda093e32138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.665400 3581 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:44.665514 3469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [term 1 FOLLOWER]: Refusing update from remote peer 277e1c84edc046bda29dfda093e32138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:25:44.666272 3470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "277e1c84edc046bda29dfda093e32138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef269d68357c4c23a12db7dadedb43cf"
I20260926 02:25:44.666803 3587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:44.670001 3470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 277e1c84edc046bda29dfda093e32138 in term 1.
I20260926 02:25:44.671348 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:44.644804 3353 master_runner.cc:186] Error getting master registration for 127.12.244.253:34955: OK, instance_id { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" instance_seqno: 1790389543806067 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:25:44.672616 3353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:25:44.681488 3587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "277e1c84edc046bda29dfda093e32138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.681898 3587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:44.685555 3604 catalog_manager.cc:1347] Generated new cluster ID: 1058ba0fb53a4ea1a372f51e6b517389
I20260926 02:25:44.685717 3604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:44.726187 3629 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "277e1c84edc046bda29dfda093e32138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.726799 3629 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:44.727171 3619 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 277e1c84edc046bda29dfda093e32138. Latest consensus state: current_term: 1 leader_uuid: "277e1c84edc046bda29dfda093e32138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.727496 3619 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:44.748939 3601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "277e1c84edc046bda29dfda093e32138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "277e1c84edc046bda29dfda093e32138" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 44181 } } peers { permanent_uuid: "ef269d68357c4c23a12db7dadedb43cf" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 34955 } } peers { permanent_uuid: "aec0246d46fb458eb2c3a951ce8a7ed6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 38149 } } }
I20260926 02:25:44.749385 3601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:44.761094 3604 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:44.763728 3604 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:44.840683 3604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 277e1c84edc046bda29dfda093e32138: Generated new TSK 0
I20260926 02:25:44.842154 3604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:44.867411 3641 catalog_manager.cc:1259] Loaded cluster ID: 1058ba0fb53a4ea1a372f51e6b517389
I20260926 02:25:44.867537 3641 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf: loading cluster ID for follower catalog manager: success
I20260926 02:25:44.871469 3641 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf: acquiring CA information for follower catalog manager: success
I20260926 02:25:44.875229 3641 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef269d68357c4c23a12db7dadedb43cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:25:44.977475 3577 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:44.968526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.12.244.253:34955)
0926 02:25:44.968694 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.968700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.975262 (+ 6562us) server_negotiation.cc:407] Connection header received
0926 02:25:44.975785 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.975792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.975914 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.976006 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.977202 (+ 1196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42562: BlockingRecv error: recv got EOF from 127.0.0.1:42562 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 02:25:44.979967 3576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:44.968132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.12.244.254:44181)
0926 02:25:44.968313 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.968321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.978202 (+ 9881us) server_negotiation.cc:407] Connection header received
0926 02:25:44.978453 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.978458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.978573 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.978648 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.979691 (+ 1043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57520: BlockingRecv error: recv got EOF from 127.0.0.1:57520 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 02:25:44.984117 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:44.981409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.12.244.252:38149)
0926 02:25:44.981607 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:44.981613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:44.982249 (+ 636us) server_negotiation.cc:407] Connection header received
0926 02:25:44.982492 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:44.982498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:44.982602 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:44.982678 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:44.983838 (+ 1160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50352: BlockingRecv error: recv got EOF from 127.0.0.1:50352 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:25:45.039980 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.020502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.12.244.253:34955)
0926 02:25:45.020709 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.020717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.031625 (+ 10908us) server_negotiation.cc:407] Connection header received
0926 02:25:45.031989 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.032008 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.032272 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.032424 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.033158 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.034346 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.036495 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.037257 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.037379 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.038212 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.039026 (+ 814us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.039197 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.039464 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.039575 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:25:45.098929 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:45.085244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.12.244.252:38149)
0926 02:25:45.085481 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.085488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.096492 (+ 11004us) server_negotiation.cc:407] Connection header received
0926 02:25:45.097112 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.097119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.097266 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.097361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:45.098669 (+ 1308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50354: BlockingRecv error: recv got EOF from 127.0.0.1:50354 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:25:45.152992 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.134188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.12.244.253:34955)
0926 02:25:45.134401 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.134408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.145578 (+ 11170us) server_negotiation.cc:407] Connection header received
0926 02:25:45.146236 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.146247 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.146344 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.146412 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.147180 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.148312 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.150451 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.151244 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.151357 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.152053 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.152241 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.152315 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.152565 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.152658 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 02:25:45.209821 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:45.194401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50356 (local address 127.12.244.252:38149)
0926 02:25:45.194677 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.194686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.207796 (+ 13110us) server_negotiation.cc:407] Connection header received
0926 02:25:45.208098 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.208109 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.208237 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.208328 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:45.209505 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50356: BlockingRecv error: recv got EOF from 127.0.0.1:50356 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:25:45.246071 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.237848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.12.244.253:34955)
0926 02:25:45.238074 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.238081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.238724 (+ 643us) server_negotiation.cc:407] Connection header received
0926 02:25:45.238948 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.238954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.239060 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.239199 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.239690 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.240910 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.243568 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.244357 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.244466 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.245166 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.245353 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.245459 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.245649 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.245747 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 02:25:45.262882 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:45.260504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.12.244.252:38149)
0926 02:25:45.260688 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.260694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.261217 (+ 523us) server_negotiation.cc:407] Connection header received
0926 02:25:45.261503 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.261509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.261602 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.261673 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:45.262667 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50372: BlockingRecv error: recv got EOF from 127.0.0.1:50372 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:25:45.282292 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.274571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.12.244.253:34955)
0926 02:25:45.274823 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.274829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.275425 (+ 596us) server_negotiation.cc:407] Connection header received
0926 02:25:45.275683 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.275693 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.275810 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.275883 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.276614 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.277828 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.280016 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.280664 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.280772 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.281403 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.281582 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.281644 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.281865 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.281969 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260926 02:25:45.313774 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:45.307820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.12.244.252:38149)
0926 02:25:45.308079 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.308096 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.311987 (+ 3891us) server_negotiation.cc:407] Connection header received
0926 02:25:45.312284 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.312290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.312393 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.312524 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:45.313486 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50376: BlockingRecv error: recv got EOF from 127.0.0.1:50376 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 02:25:45.331880 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.325435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.12.244.253:34955)
0926 02:25:45.325671 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.325679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.325977 (+ 298us) server_negotiation.cc:407] Connection header received
0926 02:25:45.326144 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.326151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.326264 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.326379 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.326830 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.328128 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.329613 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.330358 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.330514 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.330927 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.331068 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.331187 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.331425 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.331520 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260926 02:25:45.340122 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:45.338101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.12.244.252:38149)
0926 02:25:45.338288 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.338294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.338665 (+ 371us) server_negotiation.cc:407] Connection header received
0926 02:25:45.338817 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.338824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.338923 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.339043 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:45.339866 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50380: BlockingRecv error: recv got EOF from 127.0.0.1:50380 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:25:45.368860 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.361276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.12.244.253:34955)
0926 02:25:45.361467 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.361473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.362031 (+ 558us) server_negotiation.cc:407] Connection header received
0926 02:25:45.362361 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.362368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.362463 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.362579 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.363190 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.364384 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.366526 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.367306 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.367460 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.368005 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.368182 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.368298 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.368436 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.368532 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 02:25:45.383046 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:45.380735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50388 (local address 127.12.244.252:38149)
0926 02:25:45.380928 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.380935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.381465 (+ 530us) server_negotiation.cc:407] Connection header received
0926 02:25:45.381661 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.381667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.381756 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.381860 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:45.382808 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50388: BlockingRecv error: recv got EOF from 127.0.0.1:50388 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:25:45.424309 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.419023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.12.244.253:34955)
0926 02:25:45.419297 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.419307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.419636 (+ 329us) server_negotiation.cc:407] Connection header received
0926 02:25:45.419868 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.419872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.419938 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.420041 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.420471 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.421268 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.422580 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.423003 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.423167 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.423575 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.423692 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.423793 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.423944 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.424023 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260926 02:25:45.432076 3578 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:45.430093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.12.244.252:38149)
0926 02:25:45.430301 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.430308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.430728 (+ 420us) server_negotiation.cc:407] Connection header received
0926 02:25:45.430884 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.430888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.430959 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.431045 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:45.431815 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50390: BlockingRecv error: recv got EOF from 127.0.0.1:50390 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:25:45.473212 3593 catalog_manager.cc:1259] Loaded cluster ID: 1058ba0fb53a4ea1a372f51e6b517389
I20260926 02:25:45.473311 3593 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: loading cluster ID for follower catalog manager: success
I20260926 02:25:45.475947 3593 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: acquiring CA information for follower catalog manager: success
I20260926 02:25:45.477818 3593 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aec0246d46fb458eb2c3a951ce8a7ed6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:25:45.505265 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.499982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.12.244.253:34955)
0926 02:25:45.500228 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.500233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.500543 (+ 310us) server_negotiation.cc:407] Connection header received
0926 02:25:45.500714 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.500719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.500790 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.500878 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.501310 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.502099 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.503496 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.503973 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.504092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.504580 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.504719 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.504797 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.504946 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.505016 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:25:45.517043 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.510780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.12.244.252:38149)
0926 02:25:45.510958 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.510963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.511538 (+ 575us) server_negotiation.cc:407] Connection header received
0926 02:25:45.511675 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.511683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.511862 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.511960 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.512411 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.513354 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.514762 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.515288 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.515397 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.515886 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.516413 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.516583 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.516625 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.516696 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 02:25:45.532312 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.524347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.12.244.253:34955)
0926 02:25:45.524617 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.524626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.525639 (+ 1013us) server_negotiation.cc:407] Connection header received
0926 02:25:45.525952 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.525957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.526041 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.526133 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.527390 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.528272 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.530034 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.530708 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.530832 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.531496 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.531640 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.531742 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.531893 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.531984 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:25:45.533183 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.524095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.12.244.254:44181)
0926 02:25:45.525061 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.525070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.525563 (+ 493us) server_negotiation.cc:407] Connection header received
0926 02:25:45.525834 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.525840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.526072 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.526655 (+ 583us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.526982 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.528151 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.530301 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.530948 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.531132 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.531697 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.532200 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.532298 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.532757 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.532850 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":683,"threads_started":1}
I20260926 02:25:45.533653 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:45.526449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.12.244.252:38149)
0926 02:25:45.526655 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:45.526661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:45.526680 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:25:45.526755 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:45.526758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:45.526820 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:45.526924 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:45.527667 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.528763 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.531475 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:45.532143 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:45.532284 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:45.532869 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:45.532994 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:45.533058 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:45.533292 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 02:25:45.533390 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 02:25:45.538319 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3353
I20260926 02:25:45.540086 3415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:45.652815 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3353
I20260926 02:25:45.675056 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3427
I20260926 02:25:45.676348 3490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:45.786159 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3427
I20260926 02:25:45.807269 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3503
I20260926 02:25:45.808475 3565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:45.918154 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3503
2026-09-26T02:25:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4524 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:25:46 dist-test-slave-qfmw krb5kdc[3745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:25:46 dist-test-slave-qfmw krb5kdc[3745](info): set up 2 sockets
Sep 26 02:25:46 dist-test-slave-qfmw krb5kdc[3745](info): commencing operation
krb5kdc: starting...
W20260926 02:25:47.118773 13267 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/test-user.keytab.
Sep 26 02:25:47 dist-test-slave-qfmw krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:25:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:25:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.244.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
Entry for principal kudu/127.12.244.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab.
I20260926 02:25:47.404033 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39379
--webserver_interface=127.12.244.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39379,127.12.244.253:36021,127.12.244.252:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:25:47.751585 3758 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:25:47.751994 3758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:47.752141 3758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:47.752283 3758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:47.759848 3758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:47.759992 3758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:47.760124 3758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:47.760212 3758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:47.760324 3758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:47.770597 3758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39379,127.12.244.253:36021,127.12.244.252:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.254.keytab
--principal=kudu/127.12.244.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.254:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.244.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:47.772042 3758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:47.773458 3758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:47.785622 3763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:47.785696 3764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:47.785717 3766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:47.787133 3758 server_base.cc:1034] running on GCE node
I20260926 02:25:47.788070 3758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:47.789743 3758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:47.791007 3758 hybrid_clock.cc:648] HybridClock initialized: now 1790389547790917 us; error 69 us; skew 500 ppm
I20260926 02:25:47.791395 3758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:47 dist-test-slave-qfmw krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:47.794778 3758 init.cc:377] Logged in from keytab as kudu/127.12.244.254@KRBTEST.COM (short username kudu)
I20260926 02:25:47.796461 3758 webserver.cc:460] Webserver started at http://127.12.244.254:40457/ using document root <none> and password file <none>
I20260926 02:25:47.797168 3758 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:47.797258 3758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:47.797546 3758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:47.800760 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data/instance:
uuid: "7528dfc3f552439293f67093178d5468"
format_stamp: "Formatted at 2026-09-26 02:25:47 on dist-test-slave-qfmw"
server_key: "df70f86798629c2dc5e5e87dd724d3dc"
server_key_iv: "a3fd83cb4419ad29ceb86ac9c69e0c94"
server_key_version: "encryptionkey@0"
I20260926 02:25:47.801576 3758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/wal/instance:
uuid: "7528dfc3f552439293f67093178d5468"
format_stamp: "Formatted at 2026-09-26 02:25:47 on dist-test-slave-qfmw"
server_key: "df70f86798629c2dc5e5e87dd724d3dc"
server_key_iv: "a3fd83cb4419ad29ceb86ac9c69e0c94"
server_key_version: "encryptionkey@0"
I20260926 02:25:47.807183 3758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 02:25:47.811012 3773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:47.812760 3758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 02:25:47.812927 3758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/wal
uuid: "7528dfc3f552439293f67093178d5468"
format_stamp: "Formatted at 2026-09-26 02:25:47 on dist-test-slave-qfmw"
server_key: "df70f86798629c2dc5e5e87dd724d3dc"
server_key_iv: "a3fd83cb4419ad29ceb86ac9c69e0c94"
server_key_version: "encryptionkey@0"
I20260926 02:25:47.813136 3758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:47.831410 3758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:47.836277 3758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:47.836536 3758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:47.856544 3758 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.254:39379
I20260926 02:25:47.856556 3824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.254:39379 every 8 connection(s)
I20260926 02:25:47.857995 3758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/data/info.pb
I20260926 02:25:47.861629 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3758
I20260926 02:25:47.861835 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-0/wal/instance
I20260926 02:25:47.862214 3825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:47.862385 13267 external_mini_cluster.cc:1428] Setting key f55ad24db248b607efcfc257fd0ef9f6
I20260926 02:25:47.869235 3825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:47 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.244.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.253@KRBTEST.COM" created.
I20260926 02:25:47.893277 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:47.872894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.12.244.254:39379)
0926 02:25:47.874657 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:47.874667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:47.874727 (+ 60us) server_negotiation.cc:407] Connection header received
0926 02:25:47.876456 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:47.876463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:47.876728 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:47.876939 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:47.878327 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:47.879613 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:47.881046 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:47.881634 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:47.881849 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:47.885404 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:47.885413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:47.885426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:47.885455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:47.888692 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:47.889640 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:47.889647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:47.889650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:47.889800 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:47.890334 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:47.890338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:47.890340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:47.890884 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:47.891076 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:47.891823 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 02:25:47.892119 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":1298,"threads_started":1}
I20260926 02:25:47.893340 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:47.872895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39379 (local address 127.0.0.1:57480)
0926 02:25:47.873817 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:47.873866 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:47.876054 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:47.876951 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:47.876957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:47.877724 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:47.878103 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:47.878121 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:47.879775 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:47.879782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:47.880895 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:47.880907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:47.881897 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:47.881904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:47.882075 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:47.882799 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:47.882823 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:47.885152 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:47.888842 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:47.888859 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:47.888876 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:47.889495 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:47.889918 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:47.889926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:47.889933 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:47.890162 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:47.891048 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:47.891059 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:47.891351 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 02:25:47.891705 (+ 354us) client_negotiation.cc:240] Negotiation successful
0926 02:25:47.892323 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":56,"thread_start_us":394,"threads_started":1}
I20260926 02:25:47.896989 3825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:47.898380 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:47.897837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44836)
0926 02:25:47.898050 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:47.898205 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 02:25:47.898661 3776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
I20260926 02:25:47.900493 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:47.900112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44846)
0926 02:25:47.900286 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:47.900357 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 02:25:47.900775 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:36021: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab.
Entry for principal kudu/127.12.244.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab.
I20260926 02:25:47.922497 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:36021
--webserver_interface=127.12.244.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab
--principal=kudu/127.12.244.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39379,127.12.244.253:36021,127.12.244.252:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:25:47.950133 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } attempt: 1
I20260926 02:25:47.951472 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:47.950889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44860)
0926 02:25:47.951152 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:47.951300 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:25:47.952625 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:47.952219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44864)
0926 02:25:47.952376 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:47.952463 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:25:47.952863 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:36021: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
I20260926 02:25:48.021180 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } attempt: 2
I20260926 02:25:48.022544 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.021902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44868)
0926 02:25:48.022215 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.022343 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 02:25:48.023825 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.023282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44872)
0926 02:25:48.023464 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.023573 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 02:25:48.024060 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:36021: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
I20260926 02:25:48.115383 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } attempt: 3
I20260926 02:25:48.116616 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.116165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44878)
0926 02:25:48.116374 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.116473 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 02:25:48.117877 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.117462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44886)
0926 02:25:48.117657 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.117729 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260926 02:25:48.118177 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:36021: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
I20260926 02:25:48.261484 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } attempt: 4
I20260926 02:25:48.262799 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.262216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44900)
0926 02:25:48.262540 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.262663 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260926 02:25:48.263911 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.263536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44912)
0926 02:25:48.263703 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.263792 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 02:25:48.264163 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.253:36021: Network error: Client connection negotiation failed: client connection to 127.12.244.253:36021: connect: Connection refused (error 111)
W20260926 02:25:48.276818 3832 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:25:48.277177 3832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:48.277318 3832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:48.277465 3832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:48.285172 3832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:48.285291 3832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:48.285387 3832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:48.285480 3832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:48.285542 3832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:48.296636 3832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39379,127.12.244.253:36021,127.12.244.252:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.253.keytab
--principal=kudu/127.12.244.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.253:36021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.244.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:48.298132 3832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:48.299674 3832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:48.312745 3837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:48.317224 3840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:48.317374 3838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:48.317374 3832 server_base.cc:1034] running on GCE node
I20260926 02:25:48.318516 3832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:48.320295 3832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:48.321566 3832 hybrid_clock.cc:648] HybridClock initialized: now 1790389548321497 us; error 48 us; skew 500 ppm
I20260926 02:25:48.321959 3832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:48.331974 3832 init.cc:377] Logged in from keytab as kudu/127.12.244.253@KRBTEST.COM (short username kudu)
I20260926 02:25:48.333743 3832 webserver.cc:460] Webserver started at http://127.12.244.253:44851/ using document root <none> and password file <none>
I20260926 02:25:48.334508 3832 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:48.334601 3832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:48.334879 3832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:48.338151 3832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data/instance:
uuid: "757c2535b9b54156a6bf92b03a2146eb"
format_stamp: "Formatted at 2026-09-26 02:25:48 on dist-test-slave-qfmw"
server_key: "301bcade8cd4210fed28d72b0d85a544"
server_key_iv: "d9dadb68130ccb89265e18b61675284d"
server_key_version: "encryptionkey@0"
I20260926 02:25:48.338994 3832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/wal/instance:
uuid: "757c2535b9b54156a6bf92b03a2146eb"
format_stamp: "Formatted at 2026-09-26 02:25:48 on dist-test-slave-qfmw"
server_key: "301bcade8cd4210fed28d72b0d85a544"
server_key_iv: "d9dadb68130ccb89265e18b61675284d"
server_key_version: "encryptionkey@0"
I20260926 02:25:48.344453 3832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:25:48.348294 3847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:48.349924 3832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:25:48.350129 3832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/wal
uuid: "757c2535b9b54156a6bf92b03a2146eb"
format_stamp: "Formatted at 2026-09-26 02:25:48 on dist-test-slave-qfmw"
server_key: "301bcade8cd4210fed28d72b0d85a544"
server_key_iv: "d9dadb68130ccb89265e18b61675284d"
server_key_version: "encryptionkey@0"
I20260926 02:25:48.350373 3832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:48.366801 3832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:48.371697 3832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:48.371994 3832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:48.391980 3832 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.253:36021
I20260926 02:25:48.392000 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.253:36021 every 8 connection(s)
I20260926 02:25:48.393533 3832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/data/info.pb
I20260926 02:25:48.397868 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:48.400599 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3832
I20260926 02:25:48.400815 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-1/wal/instance
I20260926 02:25:48.401264 13267 external_mini_cluster.cc:1428] Setting key 1a31e0f4a6fe0b25c702fd0127af8f6e
I20260926 02:25:48.405123 3900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.244.252@KRBTEST.COM; defaulting to no policy
I20260926 02:25:48.430912 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.408083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.12.244.254:39379)
0926 02:25:48.408624 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.408629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.410008 (+ 1379us) server_negotiation.cc:407] Connection header received
0926 02:25:48.412494 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.412504 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.412619 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.412767 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.414525 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.415790 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.417596 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.418488 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.418663 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.422864 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:48.422874 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:48.422892 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:48.422968 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:48.427254 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.428255 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.428265 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.428269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.428467 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.429048 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.429052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.429054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.429367 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:48.429561 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:48.430427 (+ 866us) server_negotiation.cc:299] Negotiation successful
0926 02:25:48.430636 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.244.252@KRBTEST.COM" created.
I20260926 02:25:48.432464 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.408844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39379 (local address 127.0.0.1:57494)
0926 02:25:48.409769 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.409830 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:48.412142 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:48.413005 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:48.413014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:48.413621 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:48.414244 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.414264 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.415993 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.415997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.417422 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.417436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.418755 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.418760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.418992 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.419903 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:48.419934 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:48.422535 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:48.427503 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.427513 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.427530 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.428086 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.428636 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.428641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.428644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.428875 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.429564 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:48.429592 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:48.430058 (+ 466us) client_negotiation.cc:769] Sending connection context
0926 02:25:48.430323 (+ 265us) client_negotiation.cc:240] Negotiation successful
0926 02:25:48.430872 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":355,"threads_started":1}
I20260926 02:25:48.434480 3900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
I20260926 02:25:48.451116 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.435358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44926)
0926 02:25:48.435550 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.435574 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:48.435697 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:48.436873 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:48.436877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:48.437356 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:48.437492 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.437501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.439652 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.439656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.440553 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.440566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.441438 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.441444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.441639 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.442297 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:48.442315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:48.444882 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:48.448462 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.448468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.448472 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.448859 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.449270 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.449278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.449282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.449430 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.450070 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:48.450074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:48.450215 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 02:25:48.450358 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 02:25:48.450680 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 02:25:48.451243 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.435691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.12.244.253:36021)
0926 02:25:48.436371 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.436377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.436412 (+ 35us) server_negotiation.cc:407] Connection header received
0926 02:25:48.436506 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.436513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.436719 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.436890 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.437761 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.439494 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.440705 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.441290 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.441426 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.445150 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:48.445166 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:48.445183 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:48.445217 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:48.448296 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.449020 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.449028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.449030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.449149 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.449549 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.449556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.449559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.449934 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:48.450151 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:48.450520 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 02:25:48.450797 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":534,"thread_start_us":394,"threads_started":1}
I20260926 02:25:48.454149 3900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:48.455509 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.454908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42862)
0926 02:25:48.455175 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.455322 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260926 02:25:48.455767 3852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.457525 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.457116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42876)
0926 02:25:48.457278 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.457358 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 02:25:48.457723 3900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.244.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab.
Entry for principal kudu/127.12.244.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab.
I20260926 02:25:48.467371 13267 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3VVuru/build/asan/bin/kudu
/tmp/dist-test-task3VVuru/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:44033
--webserver_interface=127.12.244.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab
--principal=kudu/127.12.244.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.244.212:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.244.254:39379,127.12.244.253:36021,127.12.244.252:44033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 02:25:48.507010 3900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 1
I20260926 02:25:48.508311 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.507751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42878)
0926 02:25:48.507987 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.508118 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 02:25:48.509399 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.508990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42888)
0926 02:25:48.509162 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.509264 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 02:25:48.509657 3900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.563429 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } attempt: 5
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
I20260926 02:25:48.577881 3900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 2
I20260926 02:25:48.578233 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.564475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.12.244.253:36021)
0926 02:25:48.564692 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.564697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.564720 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:25:48.564791 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.564795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.564862 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.564976 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.565822 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.566545 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.567863 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.568354 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.568531 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.572263 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:48.572269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:48.572272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:48.572309 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:48.575298 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.576073 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.576079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.576085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.576221 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.576642 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.576647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.576652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.576981 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:48.577156 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:48.577561 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 02:25:48.577802 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 02:25:48.578230 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.564253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44942)
0926 02:25:48.564458 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.564481 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:48.564624 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:48.565046 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:48.565050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:48.565467 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:48.565647 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.565656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.566697 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.566704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.567695 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.567708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.568608 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.568615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.568830 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.569599 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:48.569620 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:48.572012 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:48.575490 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.575505 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.575511 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.575925 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.576349 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.576355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.576359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.576521 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.577125 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:48.577130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:48.577290 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 02:25:48.577447 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 02:25:48.577827 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 02:25:48.579061 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.578504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42898)
0926 02:25:48.578789 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.578883 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260926 02:25:48.579540 3825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:48.580111 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.579745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42910)
0926 02:25:48.579880 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.579976 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 02:25:48.580636 3900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.580814 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.580361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42926)
0926 02:25:48.580544 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.580637 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 02:25:48.581959 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.581573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42942)
0926 02:25:48.581736 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.581822 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 02:25:48.582183 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.633435 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 1
I20260926 02:25:48.634775 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.634219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42954)
0926 02:25:48.634456 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.634573 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:25:48.636054 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.635562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42968)
0926 02:25:48.635732 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.635833 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:25:48.636374 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.671872 3900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 3
I20260926 02:25:48.673226 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.672589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42972)
0926 02:25:48.672882 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.673011 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 02:25:48.674593 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.674033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42984)
0926 02:25:48.674321 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.674435 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260926 02:25:48.674856 3900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.704650 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 2
I20260926 02:25:48.706087 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.705429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:42992)
0926 02:25:48.705723 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.705864 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 02:25:48.707299 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.706858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43006)
0926 02:25:48.707023 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.707148 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 02:25:48.707511 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.813746 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 3
I20260926 02:25:48.815006 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.814449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43014)
0926 02:25:48.814680 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.814798 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 02:25:48.816303 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.815847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43024)
0926 02:25:48.816041 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.816179 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 02:25:48.816614 3825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
I20260926 02:25:48.818075 3900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 4
I20260926 02:25:48.819294 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.818707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43034)
0926 02:25:48.818946 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.819050 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 02:25:48.820550 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.820081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43048)
0926 02:25:48.820255 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.820366 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 02:25:48.820806 3900 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.244.252:44033: Network error: Client connection negotiation failed: client connection to 127.12.244.252:44033: connect: Connection refused (error 111)
W20260926 02:25:48.828507 3908 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 02:25:48.828938 3908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:25:48.829067 3908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:25:48.829203 3908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:25:48.837074 3908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:25:48.837222 3908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:25:48.837350 3908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:25:48.837436 3908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:25:48.837546 3908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:25:48.848583 3908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.244.212:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.244.254:39379,127.12.244.253:36021,127.12.244.252:44033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/krb5kdc/kudu_127.12.244.252.keytab
--principal=kudu/127.12.244.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.244.252:44033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.244.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 00:48:45 UTC on ccycloud.quasar-cadlry.root.comops.site
build id 4673501
ASAN enabled
I20260926 02:25:48.850009 3908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:25:48.851600 3908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:25:48.865929 3914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:48.866154 3913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:25:48.866477 3916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:25:48.867431 3908 server_base.cc:1034] running on GCE node
I20260926 02:25:48.868332 3908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:25:48.869783 3908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:25:48.871019 3908 hybrid_clock.cc:648] HybridClock initialized: now 1790389548870945 us; error 36 us; skew 500 ppm
I20260926 02:25:48.871430 3908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:25:48.874867 3908 init.cc:377] Logged in from keytab as kudu/127.12.244.252@KRBTEST.COM (short username kudu)
I20260926 02:25:48.876505 3908 webserver.cc:460] Webserver started at http://127.12.244.252:39961/ using document root <none> and password file <none>
I20260926 02:25:48.877256 3908 fs_manager.cc:359] Metadata directory not provided
I20260926 02:25:48.877374 3908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:25:48.877720 3908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:25:48.880934 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data/instance:
uuid: "be2f42d18a3c45228cd3bf446bcb71d6"
format_stamp: "Formatted at 2026-09-26 02:25:48 on dist-test-slave-qfmw"
server_key: "67bb5952ef72f31376d07d4db652260a"
server_key_iv: "e1b67869fd4ecafe3e2e963fe605df9f"
server_key_version: "encryptionkey@0"
I20260926 02:25:48.881793 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/wal/instance:
uuid: "be2f42d18a3c45228cd3bf446bcb71d6"
format_stamp: "Formatted at 2026-09-26 02:25:48 on dist-test-slave-qfmw"
server_key: "67bb5952ef72f31376d07d4db652260a"
server_key_iv: "e1b67869fd4ecafe3e2e963fe605df9f"
server_key_version: "encryptionkey@0"
I20260926 02:25:48.887413 3908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 02:25:48.891243 3923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:25:48.892786 3908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:25:48.892969 3908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data,/tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/wal
uuid: "be2f42d18a3c45228cd3bf446bcb71d6"
format_stamp: "Formatted at 2026-09-26 02:25:48 on dist-test-slave-qfmw"
server_key: "67bb5952ef72f31376d07d4db652260a"
server_key_iv: "e1b67869fd4ecafe3e2e963fe605df9f"
server_key_version: "encryptionkey@0"
I20260926 02:25:48.893216 3908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:25:48.909616 3908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:25:48.914320 3908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:25:48.914628 3908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:25:48.934726 3908 rpc_server.cc:307] RPC server started. Bound to: 127.12.244.252:44033
I20260926 02:25:48.934746 3975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.244.252:44033 every 8 connection(s)
I20260926 02:25:48.936267 3908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/data/info.pb
I20260926 02:25:48.940447 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:25:48.944871 13267 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3VVuru/build/asan/bin/kudu as pid 3908
I20260926 02:25:48.945079 13267 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3VVuru/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790389169147784-13267-0/minicluster-data/master-2/wal/instance
I20260926 02:25:48.945657 13267 external_mini_cluster.cc:1428] Setting key 4d917378c558d9395cfa57679c780c20
I20260926 02:25:48.948698 3976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:48.969545 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.953004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.12.244.254:39379)
0926 02:25:48.954436 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.954443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.954464 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:25:48.954528 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.954533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.954633 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.954740 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.956342 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.957091 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.958830 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.959349 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.959451 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.963733 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:48.963738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:48.963742 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:48.963784 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:48.966527 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.967328 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.967334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.967339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.967473 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.967898 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.967901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.967903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.968176 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:48.968337 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:48.968908 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 02:25:48.969138 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":1158,"threads_started":1}
I20260926 02:25:48.970675 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.952912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.254:39379 (local address 127.0.0.1:57498)
0926 02:25:48.953727 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.953789 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:48.953944 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:48.954842 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:48.954848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:48.955845 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:48.956178 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.956191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.957262 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.957266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.958665 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.958678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.959654 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.959664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.959907 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.960629 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:48.960669 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:48.963508 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:48.966685 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.966699 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.966713 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.967206 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.967595 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.967602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.967605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.967768 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.968307 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:48.968316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:48.968541 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 02:25:48.968804 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 02:25:48.969377 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":309,"threads_started":1}
I20260926 02:25:48.970695 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.947756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.12.244.252:44033)
0926 02:25:48.948628 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.948641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.948690 (+ 49us) server_negotiation.cc:407] Connection header received
0926 02:25:48.950695 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.950724 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.951305 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.951664 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.953511 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.955450 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.957027 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.957850 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.957981 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.963406 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:48.963416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:48.963432 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:48.963525 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:48.966594 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.967407 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.967414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.967416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.967575 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.967967 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.967971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.967973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.968535 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:48.968860 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:48.969206 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 02:25:48.969511 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":646,"spinlock_wait_cycles":21760,"thread_start_us":438,"threads_started":1}
I20260926 02:25:48.972872 3976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.254@KRBTEST.COM
I20260926 02:25:48.988355 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.973538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.253:36021 (local address 127.0.0.1:44950)
0926 02:25:48.973706 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.973724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:48.974184 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:48.974730 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:48.974735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:48.975263 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:48.975480 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.975495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.976500 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.976504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.977647 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.977662 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.978369 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.978377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.978563 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.979270 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:48.979289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:48.981982 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:48.985761 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.985767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.985772 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.986191 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.986603 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:48.986607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:48.986611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:48.986762 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:48.987346 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:48.987351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:48.987521 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 02:25:48.987675 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 02:25:48.987977 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:25:48.988384 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.973685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.12.244.253:36021)
0926 02:25:48.973875 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.973881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.973901 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:25:48.974341 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.974347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.974460 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.974576 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.975640 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.976326 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.977802 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.978217 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.978314 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.982203 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:48.982209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:48.982212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:48.982250 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:48.985619 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.986322 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.986328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.986332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.986476 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.986875 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.986881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.986884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.987231 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:48.987387 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:48.987766 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 02:25:48.987985 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:25:48.989319 3976 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } has no permanent_uuid. Determining permanent_uuid...
I20260926 02:25:48.989833 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.974657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.12.244.254:39379)
0926 02:25:48.974821 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.974825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.974838 (+ 13us) server_negotiation.cc:407] Connection header received
0926 02:25:48.974887 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.974890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.974950 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.975025 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.976432 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.977124 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.978631 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.979261 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.979357 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.983818 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:48.983825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:48.983830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:48.983873 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:48.986926 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.987667 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.987673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.987675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.987804 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:48.988196 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:48.988200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:48.988202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:48.988520 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:48.988822 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:48.989296 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 02:25:48.989547 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 02:25:48.993841 3825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 4
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.252@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.244.253@KRBTEST.COM
Sep 26 02:25:48 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389547, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.254@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
I20260926 02:25:49.007730 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.990297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43080)
0926 02:25:48.990474 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.990489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:48.990578 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:48.991161 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:48.991166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:48.991639 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:48.991821 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.991833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.993270 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.993274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.994540 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.994556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.995598 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.995607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.995870 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:48.997506 (+ 1636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:48.997529 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:49.000558 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:49.004639 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:49.004646 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:49.004654 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:49.005047 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:49.005498 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:49.005505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:49.005510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:49.005684 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:49.006397 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:49.006404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:49.006584 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 02:25:49.006794 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 02:25:49.007270 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 02:25:49.007834 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.990685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.12.244.252:44033)
0926 02:25:48.990816 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.990820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.990852 (+ 32us) server_negotiation.cc:407] Connection header received
0926 02:25:48.990902 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.990909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.990981 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.991154 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.991972 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.993137 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.994698 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.995440 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.995574 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.000771 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:49.000777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:49.000782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:49.000817 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:49.004497 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.005177 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.005184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.005189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.005342 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.005818 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.005823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.005826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.006246 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:49.006394 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.007201 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.007476 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:25:49.010707 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.991812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.12.244.253:36021)
0926 02:25:48.991985 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.991989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.992009 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:25:48.992049 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.992052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.992102 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.992188 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.993664 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.994811 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.996390 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.997088 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.997224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.002481 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:49.002490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:49.002497 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:49.002550 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:49.006695 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.007626 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.007631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.007634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.007813 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.009003 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.009008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.009012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.009411 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:49.009695 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.010109 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.010373 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:25:49.011368 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.994399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43086)
0926 02:25:48.994635 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:48.994654 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:48.994765 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:48.995733 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:48.995738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:48.996257 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:48.996482 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.996494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.998460 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.998468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:48.999695 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:48.999708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.000521 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.000528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:49.000743 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.001375 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:49.001418 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:49.003830 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:49.008467 (+ 4637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:49.008473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:49.008476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:49.009040 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:49.009498 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:49.009503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:49.009506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:49.009692 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:49.010397 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:49.010402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:49.010593 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 02:25:49.010720 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 02:25:49.011023 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 02:25:49.012491 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:48.994431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.12.244.252:44033)
0926 02:25:48.995416 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:48.995420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:48.995437 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:25:48.995472 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:48.995475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:48.995554 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:48.996623 (+ 1069us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:48.996840 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:48.998302 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:48.999911 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.000361 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.000494 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.004079 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:49.004085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:49.004088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:49.004119 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:49.007786 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.009195 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.009200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.009204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.009354 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.009838 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.009841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.009843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.010236 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:49.011640 (+ 1404us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.011741 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.011890 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":669,"threads_started":1}
I20260926 02:25:49.013820 3976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: Bootstrap starting.
I20260926 02:25:49.017235 3976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:49.017298 3825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468: Bootstrap starting.
I20260926 02:25:49.018612 3976 log.cc:824] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:49.021307 3825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:49.022243 3976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: No bootstrap required, opened a new log
I20260926 02:25:49.022264 13267 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:25:49.022640 3825 log.cc:824] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468: Log is configured to *not* fsync() on all Append() calls
I20260926 02:25:49.026252 3825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468: No bootstrap required, opened a new log
I20260926 02:25:49.025141 3800 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7528dfc3f552439293f67093178d5468"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57510
I20260926 02:25:49.033147 3976 raft_consensus.cc:348] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.033623 3976 raft_consensus.cc:729] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be2f42d18a3c45228cd3bf446bcb71d6, State: Initialized, Role: FOLLOWER
I20260926 02:25:49.034058 3825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.034197 3976 consensus_queue.cc:260] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.034479 3825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7528dfc3f552439293f67093178d5468, State: Initialized, Role: FOLLOWER
I20260926 02:25:49.035045 3825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.035305 3985 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.035781 3985 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:49.036317 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.036724 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:49.036775 3976 sys_catalog.cc:564] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:49.037506 3825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:49.037555 3800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 02:25:49.037765 3800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:49.040576 3800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.042094 3800 leader_election.cc:288] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [CANDIDATE]: Term 1 election: Requested vote from peers 757c2535b9b54156a6bf92b03a2146eb (127.12.244.253:36021), be2f42d18a3c45228cd3bf446bcb71d6 (127.12.244.252:44033)
I20260926 02:25:49.043792 3951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7528dfc3f552439293f67093178d5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "be2f42d18a3c45228cd3bf446bcb71d6"
I20260926 02:25:49.044468 3951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [term 0 FOLLOWER]: Advancing to term 1
W20260926 02:25:49.048436 3998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 02:25:49.048574 3998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 02:25:49.050216 3908 master_runner.cc:186] Error getting master registration for 127.12.244.254:39379: OK, instance_id { permanent_uuid: "7528dfc3f552439293f67093178d5468" instance_seqno: 1790389547850847 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:25:49.062729 3908 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:25:49.063176 3951 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7528dfc3f552439293f67093178d5468 in term 1.
I20260926 02:25:49.064188 3777 leader_election.cc:302] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7528dfc3f552439293f67093178d5468, be2f42d18a3c45228cd3bf446bcb71d6; no voters:
I20260926 02:25:49.064845 3986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:25:49.065690 3986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [term 1 LEADER]: Becoming Leader. State: Replica: 7528dfc3f552439293f67093178d5468, State: Running, Role: LEADER
I20260926 02:25:49.066489 3986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.071861 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7528dfc3f552439293f67093178d5468. Latest consensus state: current_term: 1 leader_uuid: "7528dfc3f552439293f67093178d5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.072464 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:49.086225 4005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 02:25:49.089705 3779 tablet.cc:2369] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:25:49.103731 4005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:25:49.120018 3900 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } attempt: 5
I20260926 02:25:49.129282 3951 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [term 1 FOLLOWER]: Refusing update from remote peer 7528dfc3f552439293f67093178d5468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Sep 26 02:25:49 dist-test-slave-qfmw krb5kdc[3745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790389548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.244.253@KRBTEST.COM for kudu/127.12.244.252@KRBTEST.COM
I20260926 02:25:49.130463 4003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:49.140801 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.120718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.244.252:44033 (local address 127.0.0.1:43114)
0926 02:25:49.121210 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 02:25:49.121231 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:25:49.121348 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:25:49.122156 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:25:49.122161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:25:49.122650 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:25:49.122864 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:49.122877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.124574 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.124579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:49.125864 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:25:49.125880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.127144 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.127151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:25:49.127408 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.128013 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:25:49.128037 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:25:49.130924 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:25:49.137225 (+ 6301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:49.137233 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:49.137238 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:49.137788 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:49.138288 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:25:49.138293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:25:49.138296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:25:49.138538 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:25:49.139442 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:25:49.139448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:25:49.139651 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 02:25:49.139836 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 02:25:49.140247 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":4}
I20260926 02:25:49.142239 3979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.121567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.12.244.252:44033)
0926 02:25:49.121754 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.121761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.121783 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:25:49.121839 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.121844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.121920 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.123017 (+ 1097us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.123227 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.124295 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.126063 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.126698 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.126846 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.131187 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:49.131195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:49.131199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:49.131238 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:49.136881 (+ 5643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.137946 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.137951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.137954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.138126 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.138693 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.138701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.138707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.139187 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:49.139367 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.141405 (+ 2038us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.141721 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:25:49.150269 3900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb: Bootstrap starting.
I20260926 02:25:49.157370 3900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb: Neither blocks nor log segments found. Creating new log.
I20260926 02:25:49.163712 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.085210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.12.244.252:44033)
0926 02:25:49.085596 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.085603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.092718 (+ 7115us) server_negotiation.cc:407] Connection header received
0926 02:25:49.093832 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.093839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.093966 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.094052 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.106242 (+ 12190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.107458 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.119658 (+ 12200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.120367 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.120509 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.145802 (+ 25293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:49.145810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:49.145814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:49.145860 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:49.150510 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.155170 (+ 4660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.155180 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.155186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.155724 (+ 538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.159157 (+ 3433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.159163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.159192 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.159889 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:49.160281 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.162456 (+ 2175us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.162940 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260926 02:25:49.165117 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.085016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.12.244.254:39379)
0926 02:25:49.085419 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.085426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.096853 (+ 11427us) server_negotiation.cc:407] Connection header received
0926 02:25:49.097783 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.097790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.097890 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.097969 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.109935 (+ 11966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.111180 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.121720 (+ 10540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.122469 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.122599 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.141208 (+ 18609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:49.141221 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:49.141225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:49.141288 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:49.145465 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.156086 (+ 10621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.156097 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.156103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.156316 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.158203 (+ 1887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.158209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.158213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.158664 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:49.158978 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.164420 (+ 5442us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.164697 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260926 02:25:49.166290 3900 log.cc:824] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb: Log is configured to *not* fsync() on all Append() calls
W20260926 02:25:49.166900 3929 tablet.cc:2369] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:25:49.178330 3900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb: No bootstrap required, opened a new log
I20260926 02:25:49.186478 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.095116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.12.244.253:36021)
0926 02:25:49.095304 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.095310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.108529 (+ 13219us) server_negotiation.cc:407] Connection header received
0926 02:25:49.109503 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.109510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.109616 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.109694 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.126463 (+ 16769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.127613 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.138398 (+ 10785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.139131 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.139235 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.173722 (+ 34487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:25:49.173733 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:25:49.173741 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:25:49.173802 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:25:49.178660 (+ 4858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.180985 (+ 2325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.180992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.180996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.181279 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:25:49.182109 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:25:49.182137 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:25:49.182141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:25:49.182823 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:25:49.183381 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.184857 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.185488 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:25:49.204361 3900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.205142 3900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757c2535b9b54156a6bf92b03a2146eb, State: Initialized, Role: FOLLOWER
I20260926 02:25:49.206058 3900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } }
I20260926 02:25:49.207314 3985 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7528dfc3f552439293f67093178d5468. Latest consensus state: current_term: 1 leader_uuid: "7528dfc3f552439293f67093178d5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.207654 3985 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:49.215690 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.216624 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:49.217566 4015 mvcc.cc:205] Tried to move back new op lower bound from 7333435593218293760 to 7333435592985108480. Current Snapshot: MvccSnapshot[applied={T|T < 7333435593218293760}]
I20260926 02:25:49.218395 3875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7528dfc3f552439293f67093178d5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "757c2535b9b54156a6bf92b03a2146eb"
I20260926 02:25:49.219154 3900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:25:49.219043 3874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:25:49.233026 3985 sys_catalog.cc:455] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7528dfc3f552439293f67093178d5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.233449 3985 sys_catalog.cc:458] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:49.234581 3874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [term 1 FOLLOWER]: Refusing update from remote peer 7528dfc3f552439293f67093178d5468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:25:49.239104 4005 catalog_manager.cc:1347] Generated new cluster ID: 73762e8cac004ea488f84de3b6080262
I20260926 02:25:49.239660 4005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:25:49.247383 3875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7528dfc3f552439293f67093178d5468 in term 1.
I20260926 02:25:49.255833 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7528dfc3f552439293f67093178d5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.256381 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:49.257761 4030 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:25:49.258198 4003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:25:49.261828 3758 master_runner.cc:186] Error getting master registration for 127.12.244.253:36021: OK, instance_id { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" instance_seqno: 1790389548386149 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 02:25:49.303730 3758 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 02:25:49.316629 4005 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:25:49.319106 4005 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:25:49.321902 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7528dfc3f552439293f67093178d5468. Latest consensus state: current_term: 1 leader_uuid: "7528dfc3f552439293f67093178d5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.322381 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:49.323709 3986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7528dfc3f552439293f67093178d5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.324043 3986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468 [sys.catalog]: This master's current role is: LEADER
I20260926 02:25:49.322659 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7528dfc3f552439293f67093178d5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7528dfc3f552439293f67093178d5468" member_type: VOTER last_known_addr { host: "127.12.244.254" port: 39379 } } peers { permanent_uuid: "757c2535b9b54156a6bf92b03a2146eb" member_type: VOTER last_known_addr { host: "127.12.244.253" port: 36021 } } peers { permanent_uuid: "be2f42d18a3c45228cd3bf446bcb71d6" member_type: VOTER last_known_addr { host: "127.12.244.252" port: 44033 } } }
I20260926 02:25:49.326716 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 02:25:49.349063 4005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7528dfc3f552439293f67093178d5468: Generated new TSK 0
I20260926 02:25:49.350193 4005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:25:49.362443 4044 catalog_manager.cc:1259] Loaded cluster ID: 73762e8cac004ea488f84de3b6080262
I20260926 02:25:49.362653 4044 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb: loading cluster ID for follower catalog manager: success
I20260926 02:25:49.375154 4044 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb: acquiring CA information for follower catalog manager: success
I20260926 02:25:49.377938 4044 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 757c2535b9b54156a6bf92b03a2146eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 02:25:49.490741 3981 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.478335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57530 (local address 127.12.244.254:39379)
0926 02:25:49.478608 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.478615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.487659 (+ 9044us) server_negotiation.cc:407] Connection header received
0926 02:25:49.488963 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.488970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.489078 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.489170 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.490439 (+ 1269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57530: BlockingRecv error: recv got EOF from 127.0.0.1:57530 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 02:25:49.495852 3905 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.492894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.12.244.253:36021)
0926 02:25:49.493120 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.493128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.493841 (+ 713us) server_negotiation.cc:407] Connection header received
0926 02:25:49.494262 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.494269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.494368 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.494449 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.495586 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44960: BlockingRecv error: recv got EOF from 127.0.0.1:44960 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 02:25:49.498188 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.482372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.12.244.252:44033)
0926 02:25:49.482578 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.482585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.492302 (+ 9717us) server_negotiation.cc:407] Connection header received
0926 02:25:49.496229 (+ 3927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.496236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.496331 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.496409 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.497896 (+ 1487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43120: BlockingRecv error: recv got EOF from 127.0.0.1:43120 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:25:49.553160 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.533420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.12.244.253:36021)
0926 02:25:49.533657 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.533665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.543154 (+ 9489us) server_negotiation.cc:407] Connection header received
0926 02:25:49.543919 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.543926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.544176 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.544341 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:49.545210 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.546505 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.549739 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.550480 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.550587 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.551463 (+ 876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:49.552253 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:49.552385 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.552647 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.552764 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 02:25:49.596715 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.587020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.12.244.252:44033)
0926 02:25:49.587336 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.587344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.594361 (+ 7017us) server_negotiation.cc:407] Connection header received
0926 02:25:49.594956 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.594963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.595140 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.595235 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.596454 (+ 1219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43134: BlockingRecv error: recv got EOF from 127.0.0.1:43134 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260926 02:25:49.646360 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.630838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.12.244.253:36021)
0926 02:25:49.631067 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.631073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.638730 (+ 7657us) server_negotiation.cc:407] Connection header received
0926 02:25:49.639319 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.639327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.639426 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.639497 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:49.640414 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.641667 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.643903 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.644585 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.644703 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.645397 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:49.645588 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:49.645658 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.645908 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.646007 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 02:25:49.689786 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.680523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.12.244.252:44033)
0926 02:25:49.680740 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.680746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.687702 (+ 6956us) server_negotiation.cc:407] Connection header received
0926 02:25:49.688256 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.688263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.688373 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.688462 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.689529 (+ 1067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43150: BlockingRecv error: recv got EOF from 127.0.0.1:43150 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:25:49.744709 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.729161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.12.244.253:36021)
0926 02:25:49.729380 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.729387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.737271 (+ 7884us) server_negotiation.cc:407] Connection header received
0926 02:25:49.737669 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.737676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.737802 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.737880 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:49.738755 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.740084 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.742224 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.742937 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.743047 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.743772 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:49.743923 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:49.743987 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.744234 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.744331 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 02:25:49.787617 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.778534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.12.244.252:44033)
0926 02:25:49.778732 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.778738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.785619 (+ 6881us) server_negotiation.cc:407] Connection header received
0926 02:25:49.786125 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.786132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.786228 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.786301 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.787347 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43164: BlockingRecv error: recv got EOF from 127.0.0.1:43164 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:25:49.842484 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.829081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.12.244.253:36021)
0926 02:25:49.829294 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.829300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.835816 (+ 6516us) server_negotiation.cc:407] Connection header received
0926 02:25:49.836132 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.836139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.836238 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.836366 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:49.836961 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.838145 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.840397 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.841009 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.841177 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.841671 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:49.841830 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:49.841921 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.842138 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.842212 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 02:25:49.851313 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.849006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.12.244.252:44033)
0926 02:25:49.849295 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.849302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.849690 (+ 388us) server_negotiation.cc:407] Connection header received
0926 02:25:49.849899 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.849905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.850013 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.850158 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.851011 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43176: BlockingRecv error: recv got EOF from 127.0.0.1:43176 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260926 02:25:49.872030 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.864664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.12.244.253:36021)
0926 02:25:49.864854 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.864860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.865441 (+ 581us) server_negotiation.cc:407] Connection header received
0926 02:25:49.865657 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.865666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.865777 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.865893 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:49.866371 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.867634 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.869611 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.870326 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.870486 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.871014 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:49.871237 (+ 223us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:49.871339 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.871570 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.871671 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 02:25:49.883651 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.881403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.12.244.252:44033)
0926 02:25:49.881637 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.881646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.882015 (+ 369us) server_negotiation.cc:407] Connection header received
0926 02:25:49.882263 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.882268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.882359 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.882482 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.883391 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43178: BlockingRecv error: recv got EOF from 127.0.0.1:43178 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:25:49.913540 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.904039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.12.244.253:36021)
0926 02:25:49.904275 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.904284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.906396 (+ 2112us) server_negotiation.cc:407] Connection header received
0926 02:25:49.906748 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.906760 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.906865 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.906985 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:49.907529 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.909156 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.911273 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.911758 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.911879 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.912619 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:49.912777 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:49.912869 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.913083 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.913170 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 02:25:49.924122 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.921311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.12.244.252:44033)
0926 02:25:49.921546 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.921553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.922208 (+ 655us) server_negotiation.cc:407] Connection header received
0926 02:25:49.922486 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.922491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.922583 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.922698 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.923881 (+ 1183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43186: BlockingRecv error: recv got EOF from 127.0.0.1:43186 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 02:25:49.966748 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:49.960500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.12.244.253:36021)
0926 02:25:49.960755 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.960761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.961073 (+ 312us) server_negotiation.cc:407] Connection header received
0926 02:25:49.961367 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.961371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.961484 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.961580 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:49.962189 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.962942 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.964686 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:49.965198 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:49.965313 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:49.965883 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:49.966011 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:49.966093 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:49.966349 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 02:25:49.966442 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 02:25:49.975402 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:49.973235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.12.244.252:44033)
0926 02:25:49.973504 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:49.973511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:49.973855 (+ 344us) server_negotiation.cc:407] Connection header received
0926 02:25:49.974092 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:49.974098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:49.974173 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:49.974283 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:49.975077 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43202: BlockingRecv error: recv got EOF from 127.0.0.1:43202 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 02:25:50.048396 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:50.043287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.12.244.253:36021)
0926 02:25:50.043547 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:50.043553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:50.043874 (+ 321us) server_negotiation.cc:407] Connection header received
0926 02:25:50.044042 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:50.044046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:50.044122 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:50.044207 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:50.044622 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.045374 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.046712 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.047177 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.047279 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:50.047669 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:50.047788 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:50.047859 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:50.047995 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 02:25:50.048059 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 02:25:50.050850 3998 catalog_manager.cc:1259] Loaded cluster ID: 73762e8cac004ea488f84de3b6080262
I20260926 02:25:50.050973 3998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: loading cluster ID for follower catalog manager: success
I20260926 02:25:50.054136 3998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: acquiring CA information for follower catalog manager: success
W20260926 02:25:50.057144 3982 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:25:50.054849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.12.244.252:44033)
0926 02:25:50.055163 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:50.055169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:50.055566 (+ 397us) server_negotiation.cc:407] Connection header received
0926 02:25:50.055796 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:50.055800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:50.055867 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:50.055954 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:25:50.056864 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43206: BlockingRecv error: recv got EOF from 127.0.0.1:43206 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":110}
I20260926 02:25:50.057153 3998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be2f42d18a3c45228cd3bf446bcb71d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 02:25:50.198189 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:50.192693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.12.244.253:36021)
0926 02:25:50.192962 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:50.192969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:50.193315 (+ 346us) server_negotiation.cc:407] Connection header received
0926 02:25:50.193555 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:50.193559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:50.193637 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:50.193742 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:50.194173 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.194912 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.196421 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.196896 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.197045 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:50.197465 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:50.197582 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:50.197681 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:50.197836 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 02:25:50.197918 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 02:25:50.210307 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:50.204475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.12.244.252:44033)
0926 02:25:50.204743 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:50.204750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:50.204974 (+ 224us) server_negotiation.cc:407] Connection header received
0926 02:25:50.205163 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:50.205167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:50.205338 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:50.205439 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:50.205804 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.206553 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.208076 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.208485 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.208593 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:50.209097 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:50.209675 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:50.209790 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:50.209901 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 02:25:50.209980 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260926 02:25:50.227418 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:50.217222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.12.244.252:44033)
0926 02:25:50.217436 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:50.217443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:50.220873 (+ 3430us) server_negotiation.cc:407] Connection header received
0926 02:25:50.221335 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:50.221340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:50.221440 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:50.221537 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:50.222318 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.223212 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.225130 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.225831 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.225988 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:50.226449 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:50.226627 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:50.226717 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:50.226990 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:25:50.227062 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:25:50.227962 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:50.218413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.12.244.253:36021)
0926 02:25:50.218676 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:50.218685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:50.221228 (+ 2543us) server_negotiation.cc:407] Connection header received
0926 02:25:50.221365 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:50.221370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:50.221454 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:50.221537 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:50.222237 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.223216 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.225547 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.226261 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.226482 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:50.227014 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:50.227185 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:50.227293 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:50.227457 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 02:25:50.227712 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:25:50.228189 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:25:50.217118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.12.244.254:39379)
0926 02:25:50.220094 (+ 2976us) server_negotiation.cc:206] Beginning negotiation
0926 02:25:50.220099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:25:50.220118 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:25:50.220218 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:25:50.220222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:25:50.220440 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:25:50.220608 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:25:50.221387 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.222512 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.224895 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:25:50.225454 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:25:50.225585 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:25:50.226583 (+ 998us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:25:50.227050 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:25:50.227193 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 02:25:50.227781 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 02:25:50.227876 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2863,"thread_start_us":2759,"threads_started":1}
I20260926 02:25:50.233676 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3758
I20260926 02:25:50.234915 3820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:50.349524 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3758
I20260926 02:25:50.370900 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3832
I20260926 02:25:50.372164 3895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:50.481515 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3832
I20260926 02:25:50.498899 13267 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task3VVuru/build/asan/bin/kudu pid 3908
I20260926 02:25:50.500126 3971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:25:50.608644 13267 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3VVuru/build/asan/bin/kudu with pid 3908
2026-09-26T02:25:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4690 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9214 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (381414 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST